Latest / The Adults In The AI Room / Amar Goel on A.I. Tools, Ground Truth in LLMs, the Bito Journey and More - Amar Goel - The Earley AI Podcast with Seth Earley - Episode #036

Transcript

Amar Goel on A.I. Tools, Ground Truth in LLMs, the Bito Journey and More - Amar Goel - The Earley AI Podcast with Seth Earley - Episode #036 · The Adults In The AI Room

• 0:01
  SPEAKER_03: This is the Early AI Podcast, hosted by Seth Early and Chris Featherstone.
• 0:06
  SPEAKER_03: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:15
  SPEAKER_03: The Early AI Podcast is sponsored by Early Information Science, their digital transformation journey with design and deployment of innovative technology solutions, as well as CMS Wire.
• 0:25
  SPEAKER_03: Now, enjoy the show.
• 0:31
  SPEAKER_01: All right, terrific.
• 0:33
  SPEAKER_01: All right.
• 0:34
  SPEAKER_01: Well, welcome to the Early AI Podcast.
• 0:37
  SPEAKER_01: I'm Seth Early.
• 0:39
  SPEAKER_00: And I'm Chris Featherstone.
• 0:40
  SPEAKER_01: And we're very excited to introduce our guest for today, and we're going to discuss the new wave of AI tools, ways to incorporate ground truth in things like large language models, which is very important, how AI developer tools are really going to change the industry.
• 0:59
  SPEAKER_01: And we're going to head on a number of other topics.
• 1:01
  SPEAKER_01: Our guest has been a trailblazer in the world of technology and entrepreneurship.
• 1:06
  SPEAKER_01: He has an impressive track record spanning two decades.
• 1:10
  SPEAKER_01: He's currently the founder and CEO of BidO, where he continues to shape the future of the emerging AI industry.
• 1:17
  SPEAKER_01: For the past 17 years, he served as the founder and chairman of PubMatic, which was a public company, establishing it as a powerhouse in the digital advertising landscape.
• 1:28
  SPEAKER_01: He's also made contributions as board member for Phoenixel, and he's brought a wealth of experience to the table.
• 1:37
  SPEAKER_01: With a terrific career like this, we're really honored to have Amr Goyle, a true uh leader in the tech world on the show today.
• 1:46
  SPEAKER_01: Amr, uh welcome to the show.
• 1:49
  SPEAKER_04: It's nice to nice to be on the show.
• 1:52
  SPEAKER_04: Um, and uh I'm kind of laughing as you're reading the bio.
• 1:54
  SPEAKER_04: I think you know my wife would have a very different bio.
• 1:57
  SPEAKER_01: So I know it's always hard when somebody's reading your bio, you're like, hmm, maybe I should read it out loud to myself, right?
• 2:06
  SPEAKER_04: Or when you're working, but excited to excited to be here and a student of AI like so many of us now.
• 2:12
  SPEAKER_04: So okay.
• 2:13
  SPEAKER_01: Well, I want to start off with a question that I usually ask our guests, and that is what are the myths and misconceptions about AI today?
• 2:22
  SPEAKER_01: So it's moving so fast, it's moving so quickly, and uh it's really hard to kind of keep up with this.
• 2:29
  SPEAKER_01: And what's really surprising to me is really how quickly some of the organizations we're working with are adopting some form of generative AI, uh, even with some of the problems that we have with large language models and Chat GPT.
• 2:44
  SPEAKER_01: But tell me, what are the things that um that you see as the biggest misconceptions or the biggest fallacies in the in the space?
• 2:52
  SPEAKER_04: Yeah, I mean, there's there's definitely a couple.
• 2:55
  SPEAKER_04: Um so I think the first I would say is uh if you think back to, let's say, when Chat GPT came out in I think November of last year, um, you know, it was kind of mind blowing for a lot of people.
• 3:11
  SPEAKER_04: And it was, you know, obviously it was it was new for the vast majority of you know people.
• 3:15
  SPEAKER_04: And um it was it was really amazing.
• 3:19
  SPEAKER_04: And every day you would see something in the headlines on Twitter, you know, some demo where somebody would say, Oh, I just rigged this up, you know, over the weekend, and it would look incredible.
• 3:30
  SPEAKER_04: Like, oh my God, I could do, I could make a well, I mean, this is more of a you know, Dali type thing, but you know, I could make this picture, or you know, I just made a video, you know, based on this thing, or you know, where we help developers, and so it was like, oh, I can make this thing where you can now like write an entire program, you know, in five minutes, or um, you know, writers were showing all kinds of like, you know, I just wrote a book, you know, in uh uh using Chad GT.
• 3:55
  SPEAKER_04: Today, yeah, yeah, exactly.
• 3:57
  SPEAKER_01: Two years, and today it took me a day.
• 4:00
  SPEAKER_04: So yeah, exactly, exactly, you know, and so I think it seemed at the time like, oh my god, every everything's gonna change overnight, you know, and there was this whole idea like, oh my god, everyone's gonna lose their job and all this kind of stuff.
• 4:15
  SPEAKER_01: And I think if you like fast forward that's what I thought that's what I thought when I first saw it, I was like, oh, is that the end of uh knowledge architecture and information architecture and all the stuff I've been doing for 25 years?
• 4:27
  SPEAKER_01: And found out that yeah.
• 4:30
  SPEAKER_04: I mean, I have like I'm on tech threads with like, you know, other like tech CEOs and whatever, and people are like, this is gonna change everything.
• 4:39
  SPEAKER_04: You don't even need software anymore, and everyone's gonna lose their job by the end of next year, and other people are like, what are you talking about?
• 4:46
  SPEAKER_04: And you know, it's like totally huge, huge differences of opinion, right?
• 4:51
  SPEAKER_04: Um, and so if you fast forward here, we're not quite a year past Chat GPT, but almost right, 10 months, 11 months.
• 5:00
  SPEAKER_04: Um, in some ways, I would say like everything's changed and nothing's changed, you know.
• 5:04
  SPEAKER_04: I mean, if you look at like how you're still doing podcasts, Seth, you're not out of a job, you know.
• 5:11
  SPEAKER_04: Um, I'm still working my butt off trying to make software, you know, like it turns out that like making a demo is really easy, but actually then making a real product is really still quite hard, you know, that people want to use and that actually, you know, does what you what you care about, you know.
• 5:31
  SPEAKER_04: And I think um, so I think turning these kind of quick demos, you know, these quick ideas, oh, this prototype into a real product that actually repeatedly reliably works, you know, I think has proved to be, you know, non-trivial, you know.
• 5:49
  SPEAKER_00: Um you know, Amar, sorry, I was I was just uh letting Seth know that I had a thought for you too.
• 5:56
  SPEAKER_00: I'd love to get a sense also because we see this get announced, you know, around the beginning of the year, and then the flurry of activity and then everybody pivots to this idea that that's AI, right?
• 6:11
  SPEAKER_00: GBT.
• 6:13
  SPEAKER_04: I'm an AI, everything is AI now.
• 6:15
  SPEAKER_04: Yeah, yeah.
• 6:16
  SPEAKER_00: That's that's and and broadly because it was released to consumers and everybody was like, no, we've been doing this a long time.
• 6:22
  SPEAKER_00: Uh how much time do you find where you're having to re-educate in addition to educate, right?
• 6:30
  SPEAKER_00: In terms of pulling them back.
• 6:31
  SPEAKER_00: Okay, hold on, guys, don't get ahead of yourself.
• 6:34
  SPEAKER_00: Let me pull you back, let me reorient you on where we're at, and then let's get you, you know, you know, pointed in the right direction.
• 6:42
  SPEAKER_00: Do you find yourself doing that quite a bit?
• 6:44
  SPEAKER_00: I find myself doing that, so that's what I was asking.
• 6:46
  SPEAKER_04: Yeah, no, I think, I mean, um definitely, because I I think one of the things we've seen is that a lot of people just haven't had the time and energy to like get deep into this, you know, and so they're they're kind of like, I mean, they read the headlines, but like in terms of really understanding like what's happening or what's possible that isn't, you know, something that they've that they've you know uh really done.
• 7:16
  SPEAKER_04: So I mean, I I have a I have a friend who runs a software company and you know, he was like, Hey, I haven't really spent time on this, you know.
• 7:23
  SPEAKER_04: And and and part of it's because it's not like it's not clearly obvious in like two minutes that it's gonna like turn his business upside down or that there's some crazy big opportunity.
• 7:33
  SPEAKER_04: So, so it's just sort of like, yeah, I haven't really spent any time on this yet, you know.
• 7:37
  SPEAKER_04: And hey, what is this all about?
• 7:39
  SPEAKER_04: You know, um, it's a you know, I think there's a certain element and people are like, is this like another crypto moment?
• 7:45
  SPEAKER_04: You know, I mean, I hear that come up a lot where they were like, yeah, crypto, you know, we can argue about whether it's a fad or not.
• 7:52
  SPEAKER_04: But you know, there's definitely a sentiment that like it kind of rose and then crashed.
• 7:56
  SPEAKER_04: And so, hey, that was a fad, and that was like just whatever, some random thing.
• 8:01
  SPEAKER_04: Is this just another kind of Silicon Valley's gone run amook and just kind of doing its thing?
• 8:06
  SPEAKER_04: And then, you know, now we'll return back to reality.
• 8:09
  SPEAKER_04: So, so I think there is like um a certain amount of you know, education and re-education where you know, some people just haven't really gone deep on it yet, you know.
• 8:19
  SPEAKER_04: And it is, it is a it is a new stack, you know, it's a new tech stack, and it requires it's not like rocket science, you know, to like understand it maybe from like a you know, just a usability standpoint.
• 8:33
  SPEAKER_01: Computer science.
• 8:35
  SPEAKER_04: Yeah, you know, but so exactly, but but you know, it takes it takes it takes a minute, and you know, you gotta you gotta it's like as any new stack, you know, there's like new frameworks, new approaches, new technologies, you know, rag uh retrieval augmented generation is coming up in a big way.
• 8:53
  SPEAKER_04: So it's like, well, what does that mean?
• 8:54
  SPEAKER_04: You know, so there's just all kinds of different components that that people are people are thinking about.
• 8:59
  SPEAKER_04: So that that would be sort of, you know, kind of the the the first thing I've felt I've noticed is just that like making that demo, you know, is really easy, but then actually turning it into like a real product is is hard.
• 9:15
  SPEAKER_04: And I think a lot of right now I feel like the quiet, you know, it's a little quieter now.
• 9:21
  SPEAKER_04: I mean, there's still announcements and all that, but it's not quite as buzzy as it was, you know, nine months ago.
• 9:26
  SPEAKER_04: And I think that's just because like takes a lot of work and you got to like throw great product instincts at it and engineers and you know, build it.
• 9:35
  SPEAKER_04: And I think there were another related point to that would be if you're building for the enterprise, you know, it can't be a toy, it's gotta really work, you know, and people want, you know, people want repeatable results, right?
• 9:49
  SPEAKER_04: So like think about, you know, for us, like we're we're, you know, like I said, working to help software developers uh, you know, use these tools to do things like write code or write test cases or debug their code.
• 10:01
  SPEAKER_04: So let's say you like ask, you know, you ask um Bitto, uh, you take some code and you say, Hey, can you tell me if this code is secure?
• 10:11
  SPEAKER_04: Any recommendations to make it more secure?
• 10:13
  SPEAKER_04: Well, when you ask that question twice, you kind of want the same answer twice.
• 10:17
  SPEAKER_04: I mean, you don't, you're not okay with it, like one time telling you like do ABC, and then the other time being like do one, two, three, you know, and then you're like, Well, but isn't ABC an issue?
• 10:30
  SPEAKER_04: Like, I thought you were gonna mention that because you said it before.
• 10:33
  SPEAKER_04: So, I mean, those kinds of things matter, you know, in that enterprise, you know, context, or of course, safety, moderation, all those things come up as well.
• 10:40
  SPEAKER_01: Right.
• 10:41
  SPEAKER_01: Well, I found that um executives and many stakeholders had a lot of misconceptions about AI before Chat GPT, right?
• 10:52
  SPEAKER_01: So there's already a lot of level setting that had to be done.
• 10:56
  SPEAKER_01: There was already a lot of misconceptions in the C-suite.
• 10:59
  SPEAKER_01: But then you also have parts of organizations that have gone through a very rapid uh maturity, uh uh, you know, evolution.
• 11:08
  SPEAKER_01: And you know, we're talking with one large bank uh recently, and um they had some very sophisticated questions.
• 11:16
  SPEAKER_01: And what we were introducing to them was retrieval augmented generation, because that will eliminate hallucinations if you do it correctly, turn on the temperatures so that's not uh uh creative.
• 11:28
  SPEAKER_01: And then the idea is to say what's a creative result versus what is a uh factually uh inaccurate result, right?
• 11:37
  SPEAKER_01: Creativity is fine, you know, we have to be factually correct and factually accurate.
• 11:42
  SPEAKER_01: And then designating a particular data source and saying, if you don't have the information from this data source, say I don't know, then that stops the LLM from using its own you know language uh understanding from extrapolating and creating something right, right, totally.
• 12:00
  SPEAKER_01: Yeah, yeah.
• 12:02
  SPEAKER_01: So I think that there's a lot of education that needs to happen.
• 12:05
  SPEAKER_01: And I think that um you know organizations are still kind of stuck on this hey, is this the next magic bullet?
• 12:13
  SPEAKER_01: And you know, and do we have to uh uh curate our information?
• 12:18
  SPEAKER_01: Do we have to have a knowledge base?
• 12:19
  SPEAKER_01: When I started talking about knowledge architecture, and I actually have a result of a study where we found that the uh using a knowledge architecture um increased the uh validity or the the accuracy of answers uh from 53% to 83%.
• 12:37
  SPEAKER_01: Without the knowledge architecture, it was 53% accurate, but with it, it was 83%.
• 12:42
  SPEAKER_01: And what that did was it helped to contextualize those answers and provide you know the the uh confidence that those answers were correct.
• 12:51
  SPEAKER_01: So it still comes down to knowledge and content and data and data quality, right?
• 12:56
  SPEAKER_01: And no matter how many times people kind of hit on that theme, there's still that that um belief out there in the marketplace that this stuff is going to obviate the need for um uh data curation or or data hygiene or content curation or knowledge hygiene and so on.
• 13:14
  SPEAKER_01: So are you seeing that and how are you kind of mitigating that or addressing that when you when you come across it?
• 13:21
  SPEAKER_04: You mean just where people are kind of unclear about the need for this whole like pipeline of information, if you will?
• 13:27
  SPEAKER_01: Right.
• 13:27
  SPEAKER_01: So the the whole structure of the reference architecture, the ontologies, the metadata, you know, what what is your what is your experience there?
• 13:35
  SPEAKER_01: Because even a lot of vendors are not quite understanding that.
• 13:38
  SPEAKER_01: You know, I was interviewing one vendor uh for uh an article and they said, uh, oh, we don't need any of that.
• 13:44
  SPEAKER_01: We don't need taxonomies, we don't need ontology, we don't need metadata.
• 13:47
  SPEAKER_01: And I said, well, how what do you do with the data?
• 13:50
  SPEAKER_01: And they go, well, we have to do some data labeling.
• 13:52
  SPEAKER_01: And it's like, duh, data labeling is metadata.
• 13:55
  SPEAKER_01: Yeah, yeah, yeah.
• 13:57
  SPEAKER_04: No, I mean, I think I think, you know, we um we launched a product recently where we um built we deploy a vector database on your machine and we index all of your code and build embeddings.
• 14:09
  SPEAKER_04: And I think we're finding there that you know there's a lot of complexity to do that well.
• 14:16
  SPEAKER_04: Um one of our one of our um, one of my co-founders who runs product was like, we could probably just build a company around this.
• 14:23
  SPEAKER_04: You know, there's so much complexity around just understanding, you know, I mean, you could say language broadly, but like we we kind of mostly live in the world of code and code has its own nuances and you know, like you said, it has a you know ontology and you have a taxonomy, every language is a little bit different.
• 14:40
  SPEAKER_04: What's the structure, what understanding what's a function name, a class, you know, variables, all those kinds of things, which which matter, right?
• 14:45
  SPEAKER_04: When a when a developer is asking a question or trying to do something, if you can bring that appropriate context and you know, an understanding of the of the code to bear, then you can provide a much better result.
• 14:57
  SPEAKER_04: And you know, and you want to provide that context to the LLM, you know, to be able to, you know, help it inform it in a in the appropriate way.
• 15:05
  SPEAKER_04: So, I mean, we're we're definitely spending a lot of energy, you know, on those um kinds of kinds of things.
• 15:12
  SPEAKER_04: Or I'll give you like another um example um where um, you know, building an agent that allows you to like kind of explain a code base, uh, document a code base.
• 15:25
  SPEAKER_04: And, you know, one of the things that like one of our, you know, originally we started our very first version was just basically kind of going through every file of the of the in, let's say you have like a folder, going through every file in there and just passing it to the you know, LLM and saying, explain this, you know.
• 15:43
  SPEAKER_04: And that that's like a good start.
• 15:45
  SPEAKER_04: Um, but then people were like, but that's not really how you navigate a code base, right?
• 15:49
  SPEAKER_04: I want to really follow the code flow, you know, of like, okay, this is the main module, this is the entry point, then where you go.
• 15:56
  SPEAKER_04: And so again, it's kind of like, well, now you got to understand how that code flows, these are the functions first, and then then you if you follow down this path, you you would then go visit, you know, these functions.
• 16:07
  SPEAKER_04: And so, you know, it's understanding, you know, the organization of that of that data set, if you will, and then bringing that to bear.
• 16:15
  SPEAKER_04: Um, so I think you know, all of those points you're mentioning are are super relevant.
• 16:22
  SPEAKER_04: And I think if you go back to what I was saying earlier, like in the quick 10-minute demo that people want to make, it's like you just ignore all of that stuff, right?
• 16:32
  SPEAKER_04: You just ignore all of that complexity and you pass a function to the LLM and you're like, oh, it explained it, we're good to go, you know.
• 16:40
  SPEAKER_04: And it's like, well, that's not the most useful thing.
• 16:43
  SPEAKER_04: So so I think that there's like a lot of a lot of complexity, you know, here.
• 16:48
  SPEAKER_04: So it really is.
• 16:49
  SPEAKER_01: There really is.
• 16:50
  SPEAKER_01: And I I think you really hit the nail on the head when you talked about the fact that it's so easy to demonstrate a concept, but going from that demonstration of the concept to something that's truly production ready, you know, that's that can be a half a million dollar project, right?
• 17:04
  SPEAKER_01: Yeah, yeah, totally.
• 17:07
  SPEAKER_04: Actually, there's one other thing I would want to mention you since you just mentioned money.
• 17:10
  SPEAKER_04: Um, you know, AI AI handy stuff.
• 17:14
  SPEAKER_01: Sorry, handy, it's handy stuff, money.
• 17:17
  SPEAKER_04: Yeah, yeah, yeah, exactly.
• 17:19
  SPEAKER_04: It's useful.
• 17:20
  SPEAKER_04: Um, AI costs are not insignificant, you know.
• 17:23
  SPEAKER_04: So um, you know, you look at different models, and you know, they their costs can be really, you know, in insignificant.
• 17:33
  SPEAKER_04: And on one hand, you have kind of, I guess it's not Moore's Law.
• 17:39
  SPEAKER_04: I mean, you got Moore's Law working for you, you know, so the cost of these chips are going to keep getting cheaper, um, the GPUs, but but you're also seeing, you know, the the LLM providers themselves do a lot of optimizations, right?
• 17:52
  SPEAKER_04: To make their, you know, inference engines like more powerful or more, you know, more uh cost effective.
• 17:59
  SPEAKER_04: So you so you're getting those benefits, but at the same point, new capabilities keep getting released, right?
• 18:04
  SPEAKER_04: And so it's a little bit like, well, if you stay at the cutting edge, your costs don't really come down.
• 18:08
  SPEAKER_04: But for the same thing, you know, the costs are coming down.
• 18:11
  SPEAKER_04: And so, but but the other point I really want to make is that the the cost to run these models to to perform outputs is is not insignificant.
• 18:20
  SPEAKER_04: And and so what I mean, what I mean by that is like if you look at GPT 3.5, you know, let's just say that costs X per thousand tokens for a second, well um running GPT 4 is like basically about um uh 30 times more expensive, you know.
• 18:42
  SPEAKER_04: Yeah, so a thousand tokens of GPT four is 30 times more expensive than that.
• 18:48
  SPEAKER_04: And and just to put that into into dollars and cents, um, you know, a thousand tokens, which would be about, let's say, 3,000 characters, you know, that costs about four cents, you know, the input and the output to process that.
• 19:01
  SPEAKER_04: So on one hand, you say, ah, three cents, you know, I mean, four cents, three thousand characters.
• 19:05
  SPEAKER_04: But, you know, if you gave it if you gave it like a 10-page book, you know, I mean a 10-page section of a book, and you said, Hey, could you like summarize this um, you know, in a page, I mean, that might cost you like a dollar, you know?
• 19:21
  SPEAKER_04: Um, and so, you know, maybe as a consumer, you're like, well, I did that a couple of times a month, no big deal.
• 19:26
  SPEAKER_04: But for the the company that's like maybe doing this thousands of times a month, you know, those those costs are significant.
• 19:31
  SPEAKER_04: And so, you know, a lot of people are like, hey, I want to include this in my product, I want to give this away for free.
• 19:37
  SPEAKER_04: It's not so easy, you know, to do that.
• 19:40
  SPEAKER_04: And I think that's why you're seeing a lot of companies are saying, hey, if you want the AI capabilities to add on as our for our tool, you know, that's an extra five, 10, 20, 30 dollars, you know.
• 19:51
  SPEAKER_04: Um, and so I think it's not that it's like uh it's a non-solvable problem.
• 19:57
  SPEAKER_04: I think my my point is just like it's not like compute and storage costs, where for the average app out there, it's a couple cents a user a month or less, you know, where you don't really, you know, think about it.
• 20:09
  SPEAKER_04: You would just be like, oh, if I could get another million users, that would be amazing.
• 20:12
  SPEAKER_04: It's gonna cost me a couple grand a month.
• 20:13
  SPEAKER_04: If you got another million users on your AI app, it might cost you a hundred grand a month, you know, and so, or maybe half a million dollars a month, depending on what they're doing.
• 20:20
  SPEAKER_04: So, and then even just getting capacity to some of these high-end models isn't, you know, is is is is hard.
• 20:26
  SPEAKER_04: If you look at OpenAI GPT-4, they have an 8K model and a 32K model, not to go too deep in the weeds, but the 8K is 8,000 tokens, you know, that you can handle input and output.
• 20:37
  SPEAKER_04: And then there's a 32K model.
• 20:39
  SPEAKER_04: The 8K model, you know, came out, I think, whatever, in March, and they announced that 32K.
• 20:45
  SPEAKER_04: Most people don't have access to the 32K model, like six months later, you know.
• 20:50
  SPEAKER_04: So, like in theory, it's available, but in reality, it's not really available, you know, and so all of these things also limit what you can do and and and whatnot.
• 21:01
  SPEAKER_04: Now, the Anthropic has some amazing models, their Claude 2 model has 100,000 tokens, context tokens.
• 21:06
  SPEAKER_04: So you're like, oh, well, let me try that out.
• 21:08
  SPEAKER_04: And it's one quarter the cost of GPT-4.
• 21:11
  SPEAKER_04: So, you know, anyways, uh, my point is that there's a lot of complexity in managing these models, managing your costs, thinking about capacity, throughput, context length, which is kind of a new paradigm.
• 21:23
  SPEAKER_04: It's a little bit uh, we were just kind of talking about this internally yesterday.
• 21:26
  SPEAKER_04: Um it's a little bit like when cell phone plans first came out, and you're like, how many minutes do you get?
• 21:32
  SPEAKER_04: You know, and it was like, oh, you get a hundred minutes a month, you know, and then, or like when the AOL came out, whatever, 25 years ago, you know, you got so many minutes of dialogue time, right?
• 21:42
  SPEAKER_04: And so that's kind of what we're in right now.
• 21:44
  SPEAKER_04: And I'm sure 10 years from now, we'll be in sort of like whatever, man.
• 21:48
  SPEAKER_01: This is just like it's like water and you know, data plans and the data plans got complex and tricky.
• 21:54
  SPEAKER_01: Anyway, Chris, you had a question?
• 21:56
  SPEAKER_00: Well, well, yeah.
• 21:57
  SPEAKER_00: I mean, first just a comment.
• 22:00
  SPEAKER_00: Maybe maybe we'll get rollover tokens at some point, right?
• 22:03
  SPEAKER_00: Then we can go to the next month.
• 22:05
  SPEAKER_04: We actually, it's funny, we wrote that in our we started launched the paid plan, and we said, Oh, you get like a hundred GBT4 requests, which users don't love so far, but we said they don't roll over.
• 22:14
  SPEAKER_04: We I need to roll them over, you know?
• 22:16
  SPEAKER_00: So yeah, I'd love to get a sense too, since you guys are you know deep in the in the developer community, because right now all these code companions are just, hey, let me see what you wrote and let me look forward a bit, you know, a few lines.
• 22:29
  SPEAKER_00: We do it from the commenting side, but we're missing like good core refactoring, we're missing good coding concepts.
• 22:35
  SPEAKER_00: Because, like, for instance, um, and I'll get to my question.
• 22:38
  SPEAKER_00: We see uh a lot of organizations that are like stepping into it and just doing the baseline.
• 22:44
  SPEAKER_00: However, they don't know what good well-formed code is yet, right?
• 22:49
  SPEAKER_00: And coding practices that are good because there's also a lot of semantics.
• 22:54
  SPEAKER_00: And then you get code migration.
• 22:55
  SPEAKER_00: I'd love to get your sense around just in terms of of um, I mean, there's large language models, there's code migrations, there's code modernization techniques now.
• 23:06
  SPEAKER_00: How are you seeing all those things kind of come together with you know, with just the notion of what's out there, how do we get legacy to new?
• 23:13
  SPEAKER_00: And then what do we do from the perspective of you know, contemporary, like what I'm doing real time, right?
• 23:18
  SPEAKER_00: Do we see those things converging in your mind?
• 23:21
  SPEAKER_00: Yeah, I mean, I think um I think that's there's a world.
• 23:26
  SPEAKER_04: Yeah, that's a lot like old stuff.
• 23:29
  SPEAKER_04: Trying to think how to organize, unpack that.
• 23:31
  SPEAKER_04: Yeah, I mean, there's I think there's room for AI to help in all of these things.
• 23:34
  SPEAKER_04: So let me start a little bit of a high level and then I'll you know come down into kind of some of the specifics.
• 23:38
  SPEAKER_04: So, you know, one of the there's been this big trend over the last, let's say, five, 10 years about shift left, you know, shift left, like everything, do it earlier in the process, right?
• 23:48
  SPEAKER_04: Like security by design, privacy by design, you know, design by design, you know, test by design, right?
• 23:55
  SPEAKER_04: And I think for the developer, they're kind of like, this is a lot.
• 23:59
  SPEAKER_04: Like, I don't really get the time to just think and write code anymore.
• 24:04
  SPEAKER_04: And I and like there's so many other things I'm supposed to be doing.
• 24:07
  SPEAKER_04: And I'm not like an expert in all of these things, you know, like I'm not a security expert.
• 24:12
  SPEAKER_04: And so then people are like, oh, let's bolt on this tool, and that'll kind of help, you know, address some of those issues.
• 24:17
  SPEAKER_04: But but I think it's kind of made the modern developer life a little bit miserable.
• 24:22
  SPEAKER_04: And and we we actually think this is like a big opportunity for AI to come and help, which is like, hey, let us kind of take some of this drudgery off your plate to free you up to kind of you know be who you want to be.
• 24:36
  SPEAKER_04: Um, and so like let us help take some of that test load off your plate, you know, let us help take some of that security load off your plate, let us help, you know, analyze all the your code for these things.
• 24:48
  SPEAKER_04: You know, um, you mentioned tech dad refactoring, you know, huge issue, you know.
• 24:54
  SPEAKER_04: Um one of uh one of our angel investors is uh Sri Shivananda, he's the CTO of PayPal, um, 10,000 engineers.
• 25:02
  SPEAKER_04: And you know, he's like, Umr, why can't we just use this to like we got a new version of React?
• 25:08
  SPEAKER_04: Like, why do I have to have like teams working on this for weeks and months to upgrade us?
• 25:13
  SPEAKER_04: You know, and and the business, of course, is like, what did I get from all that time you spent, like just upgrading the version, you know?
• 25:19
  SPEAKER_04: And it's like this soft benefits.
• 25:21
  SPEAKER_04: He's like, why can't I just put AI on this and have it do it?
• 25:24
  SPEAKER_04: You know?
• 25:24
  SPEAKER_04: And so I think there's, you know, it's absolutely you know a big opportunity to to do that, you know.
• 25:31
  SPEAKER_04: And I think we we think about like, hey, can we help provide those tools that allow you to kind of do these things?
• 25:37
  SPEAKER_04: Now I think the issue becomes like, well, when PayPal is gonna use it, you know, when a when a you know, Fortune 1000 company, I mean, that stuff better work, you know, and it better rewrite and and it better work well, right?
• 25:51
  SPEAKER_04: And so I think that's where you know the complexity really creeps in.
• 25:54
  SPEAKER_04: But I think when you just when we're talking about it at a you know, at a at a podcast level, yeah, I mean, absolutely there's that opportunity, you know, where where how do we use AI to help with these with these tools?
• 26:07
  SPEAKER_04: And I think um another maybe element I'll mention that is that you know, we're seeing developers like all across the spectrum, right?
• 26:16
  SPEAKER_04: So, so like 40% of the developers using you know uh Bido today have like over five years of experience.
• 26:23
  SPEAKER_04: So a lot of people think it's like only you know, uh, you know, new newbie developers.
• 26:28
  SPEAKER_04: Yeah, but so it definitely helps everybody, but like it definitely helps newbie developers.
• 26:32
  SPEAKER_04: I mean, I have a nine-year-old son who basically used Bido to make a Chrome extension in two minutes, you know, and he was like, Hey, I'm gonna give you a product name and I want you to make a little pop-up that searches eBay, brings back a list of all the products and ranks them by price, you know, and makes the links clickable.
• 26:53
  SPEAKER_04: And then he was like, and then it produced all the code and he ran it.
• 26:55
  SPEAKER_04: It ran.
• 26:56
  SPEAKER_04: Then he's like, Well, I don't really like this UI.
• 26:57
  SPEAKER_04: Can you center this and move it over?
• 26:59
  SPEAKER_04: And you know, rewrote it.
• 27:00
  SPEAKER_04: Now, you know, he's a bright kid, but he shouldn't be able to do that, you know, he's nine.
• 27:06
  SPEAKER_04: And so, you know, I would sort of say that like it made him like a 15-year-old developer who knew what he was doing, you know.
• 27:12
  SPEAKER_04: Um, and even if you're like a pro, it'd still take you an hour to do that, and you'd be like, How does eBay API work again or whatever, right?
• 27:20
  SPEAKER_04: So it sort of democratized that access.
• 27:22
  SPEAKER_04: So I think, and it helped, you know, wrote decent quality code.
• 27:25
  SPEAKER_04: And so I think going back to that, you know, helping be that aid for that developer to make sure they write high quality code, make sure they write secure code, make sure they, you know, properly, you know, say, hey, this code could be refactored.
• 27:36
  SPEAKER_04: Do you want to do that?
• 27:38
  SPEAKER_04: I mean, I I really think that you know, you we we should we should live in a world eventually where like all of that's just kind of getting taken care of for you, you know.
• 27:48
  SPEAKER_04: And I think um as a developer, you don't have to like maybe think about all these things.
• 27:53
  SPEAKER_04: Not to say that AI is gonna be perfect and you don't have to do anything anymore, but I think you should be able to like move at a much more rapid pace, produce much higher quality code, you know, much more secure code, you know, much more performant code.
• 28:08
  SPEAKER_02: Yeah.
• 28:08
  SPEAKER_04: Um, and so I think, you know, and and have much tighter processes, right?
• 28:12
  SPEAKER_04: Better test coverage, you know, all the related artifacts too.
• 28:15
  SPEAKER_00: Yeah.
• 28:16
  SPEAKER_00: I mean, we get into I like the ship left, you know, concept in theory.
• 28:20
  SPEAKER_00: Um, however, because it does bring the microscope into the right areas in the lens of, okay, let's focus on this.
• 28:26
  SPEAKER_00: Because where, you know, I've been classic, like as a product manager for a long time, and I I utilize these types of techniques, as you can imagine, to say, okay, let me let me, you know, create a requirements document that can potentially give you the opportunity to break it down into work items, right?
• 28:46
  SPEAKER_00: From the epics into bytes, you know, bite-sizable chunks, and then let me feed that to the work item tracker, Jira, whatever, and then and potentially stub things out.
• 28:56
  SPEAKER_00: Yeah.
• 28:57
  SPEAKER_00: However, you know, I mean, that takes a very specific special type of product manager that can do though that level andor wireframes or whatever.
• 29:04
  SPEAKER_00: But now we're getting into some of these areas around, you know, well, can I stub that out?
• 29:10
  SPEAKER_00: Should I stub that out?
• 29:12
  SPEAKER_00: How much should I give the developer to not do?
• 29:15
  SPEAKER_00: Because I I I still am a fan of I I want them to be creative beasts and just you know, yeah, just you know, get into the this the nuances of why they're building things in the way and not take away all of their creativity, yeah, which I feel like some of the product processes do.
• 29:31
  SPEAKER_04: Yeah, right.
• 29:32
  SPEAKER_04: No, and and I think it's it's a little bit about like what is the the core work that the developer is doing, and then what is sort of the you know, ancillary or add-on things.
• 29:41
  SPEAKER_04: And it's not to say those aren't important, obviously, security is important, obviously, privacy is important.
• 29:45
  SPEAKER_04: Obviously, you don't want to put secrets in your code, obviously, you know, test coverage is super important, but you know, that's I wouldn't say that for most developers, that's like the core things that they're really strong at, you know?
• 29:56
  SPEAKER_04: Um, and so all these tools that have been added on, or you know, trying to put these gates into the process are a little bit like like bolt-ons.
• 30:05
  SPEAKER_04: And so you sort of are like, all right, let me just figure it out and get it done with.
• 30:08
  SPEAKER_04: But I wouldn't say that it results in the best outcome.
• 30:11
  SPEAKER_00: Are you guys having to get into like indemnification discussions and warrants and all that kind of stuff centered around, well, if you generate code for me and it doesn't work or causes you know some type of exploit, are you right?
• 30:26
  SPEAKER_00: Yeah, yeah.
• 30:28
  SPEAKER_04: Yeah, yeah.
• 30:29
  SPEAKER_04: Are you getting into those?
• 30:31
  SPEAKER_04: It's it's come up a little bit.
• 30:32
  SPEAKER_04: I mean, it hasn't been like a huge topic of concern, um, but it it comes up.
• 30:39
  SPEAKER_04: And I do know there are some organizations that have just completely outlawed, you know, these these technologies right now.
• 30:46
  SPEAKER_04: So I was talking, I won't name any names here, but I was talking about you know, like a huge um cell phone and cellular infrastructure company, and they basically were like, we're not, we don't allow any of our, you know, we don't allow any of our developers to use any of these tools.
• 31:04
  SPEAKER_04: I was a little bit surprised because I then heard about a company that had like, you know, a hundred developers say the same thing, but that was really, really rare to me.
• 31:13
  SPEAKER_04: You know, I think the vast majority, I was talking to a uh consumer services, you know, internet company, pretty big, 3,000 software developers.
• 31:25
  SPEAKER_04: And I was talking to a senior executive there, and she said, you know, we just decided that, you know, the reward is worth the risk.
• 31:33
  SPEAKER_04: You know, like we we're you know, the this is gonna help make us so much better that we're okay with the potential, you know, risks around this.
• 31:43
  SPEAKER_04: Um, I would say we get a lot more questions about, you know, protection of my data, you know, is my is my you know, is my code being used to train a model?
• 31:55
  SPEAKER_04: What's been happening with my code?
• 31:57
  SPEAKER_04: You know, it's interesting.
• 31:59
  SPEAKER_04: People generally don't seem to care as much about their documents and their data, like numbers and whatnot.
• 32:08
  SPEAKER_04: They're like happy to plug in all kinds of SaaS tools and OAuth this account in and OAuth their Salesforce data in, and all that's sucked into some tool.
• 32:16
  SPEAKER_04: But the moment it comes to their code, you know, people are like every hair on their neck goes up, you know, and they're like, hey, what happens with my code?
• 32:25
  SPEAKER_04: Where is it stored?
• 32:26
  SPEAKER_04: How's it treated?
• 32:27
  SPEAKER_04: You know, etc.
• 32:27
  SPEAKER_04: I mean, we had a a call from like a big um top five bank.
• 32:33
  SPEAKER_04: Um, and I was surprised that they called us.
• 32:36
  SPEAKER_04: I was like, we're a crappy small startup, but um, you know, they some of their developers were using our tools, you know, for scrappy, scrappy, that's the word.
• 32:46
  SPEAKER_04: Yes, thanks.
• 32:46
  SPEAKER_04: Thanks for the thanks for the edits uh can you use AI, they'll just like augment over the seven plus production.
• 32:55
  SPEAKER_04: Um, you know, and so um, you know, they've some of their developers were using the tool, and I was like, well, how are you using it?
• 33:01
  SPEAKER_04: They're like, well, we do open source work, so we're using it for that, and we really think Biddle's cool.
• 33:04
  SPEAKER_04: And we, and so you know, we want we wanted to see about using it, you know, in our organization.
• 33:10
  SPEAKER_04: And so I was we were on this call, and you know, they were like, hey, none, no code can, no data, frankly, anything, but no code for sure can leave our you know, four walls, can't leave our you know, our cloud, our private cloud, etc.
• 33:23
  SPEAKER_04: And so I was like, well, you know, right now, like this is again was like six months ago.
• 33:27
  SPEAKER_04: I was like, you know, your code doesn't need to go to the LLM for it to like operate on it.
• 33:32
  SPEAKER_04: Now there are a number of you know pretty cool services coming up, like Amazon Bedrock and and whatnot that just went into GA a few days ago, you know, that allow you to kind of keep that the inference part in your own you know private cloud.
• 33:45
  SPEAKER_04: So so I think you know, I see this as becoming like a solved you know problem.
• 33:50
  SPEAKER_04: Like you know, Llama 2 is is pretty decent from Facebook.
• 33:54
  SPEAKER_04: Um, I don't know if it's quite there yet, but like maybe Llama 3 will be there.
• 33:58
  SPEAKER_04: So, you know, I think between a number of options are developing, whether it's you know, these tools that allow you to kind of do inference in your private cloud, whether it's standing up your own, you know, uh model, that I think a lot of these issues are going to go away around the privacy side of it.
• 34:15
  SPEAKER_04: Um, you know, I think some of the copyright type stuff, you know, maybe a little bit of open question for a little bit.
• 34:22
  SPEAKER_00: Yeah, it's it's interesting.
• 34:24
  SPEAKER_00: There's um that whole notion, like I just it feels like it's just defined irony where we're we're getting into discussions about these types of scenarios.
• 34:36
  SPEAKER_00: Their legal teams have representatives in meetings, they don't understand the technology, they're asking these questions, and yet we're talking about you know, all their infrastructure that for the last five years has been put in the cloud.
• 34:49
  SPEAKER_00: And that we had no legal discussions then.
• 34:51
  SPEAKER_00: Yeah, but now we're we're talking to you because this scares the hell out of you.
• 34:56
  SPEAKER_00: And you know, and what is the the paranoia and the FUD that you're having to work through where now these you know these folks are jumping in?
• 35:04
  SPEAKER_00: So the uh it's a bunch of it's really interesting because we've had this, you know, had to walk through some of these really interesting discussion points to your point around locking it down.
• 35:13
  SPEAKER_04: So well, and I think like I think I would agree with what you were just saying.
• 35:16
  SPEAKER_04: I think they're a little bit like we don't know what we don't know yet.
• 35:19
  SPEAKER_04: We're still trying to figure out what are the right questions to ask.
• 35:21
  SPEAKER_04: I mean, we were yeah on a call with a top five travel company um in the US, like they're you know, some developers want to use a tool, security procurement are involved, and you know, they they were there was a little bit of like, hey, what are what are the things that we do care about here, right?
• 35:37
  SPEAKER_04: And it wasn't like they they haven't done this enough yet where they have a a game plan in in you know in place.
• 35:44
  SPEAKER_04: And so, I mean, I I think I kind of get it, right?
• 35:48
  SPEAKER_04: It's new and and and so everyone's kind of learning on the job, if you will, but but it is interesting, like you said, where there's concern, and then you read those, like I think what was it, Samsung that had that article by months, you know, where some developers had put something into Chat GPT, and so you know, there's those things, of course, you know, a thousand articles come out about it, and it makes everybody freak out.
• 36:12
  SPEAKER_00: Right, you know, I'll tell you what, I think fortune favors the brave.
• 36:15
  SPEAKER_00: I'm always been a fan of that kind of stuff, and so I've actually talked to that team.
• 36:19
  SPEAKER_00: Um, and and I'm like, listen, I'm not gonna consider you guys did anything bad because I think you pioneered some stuff that you know, two historical moments.
• 36:27
  SPEAKER_00: They released the model, and then we have these pioneering experts that are like, well, hell, let's try it and see what happens.
• 36:35
  SPEAKER_00: Fail fast, right?
• 36:36
  SPEAKER_00: Yeah, and then you know, it and the the problem is is it was just, you know, there was an exploit and it was, you know, very, very um, what's it called?
• 36:44
  SPEAKER_00: Uh well, the mobs fickle, right?
• 36:46
  SPEAKER_00: Especially in the public world.
• 36:48
  SPEAKER_00: So we get all the adverse reactions to it, but it did cause a lot of people to take a step back and think about it from the fact the fact that they, you know, there's a failure there.
• 36:58
  SPEAKER_00: Oh, now what do we need to do?
• 37:00
  SPEAKER_00: So did it cause the pendulum and some knee-jerk reaction?
• 37:03
  SPEAKER_00: Absolutely.
• 37:04
  SPEAKER_00: But did it ask a lot of the same, you know, some really good questions?
• 37:07
  SPEAKER_00: Absolutely.
• 37:08
  SPEAKER_04: So, you know, I mean, I think I think all these things just you know, they they slow adoption.
• 37:14
  SPEAKER_04: And and and I don't mean that in a negative way, they just they're just like reasonable concerns that that people have.
• 37:21
  SPEAKER_04: Um, and and some companies will of course say we're not gonna use this or we need this better.
• 37:26
  SPEAKER_04: I mean, I actually think it kind of happened in the cloud too.
• 37:29
  SPEAKER_04: I mean, I remember God, I want to say like 2005 or six.
• 37:36
  SPEAKER_04: I mean, I have a friend who, you know, who's a pretty smart analyst on Wall Street, and he's like, Salesforce, what a joke, man.
• 37:44
  SPEAKER_04: What company is gonna put their data in the cloud?
• 37:47
  SPEAKER_04: You know, like those guys are ridiculous.
• 37:50
  SPEAKER_04: Um, and I was like, Oh, yeah, is that what you hear from your customers?
• 37:53
  SPEAKER_04: I mean, I I you know, I was just listening to him and taking him at his word, and it was like, clearly he was wrong, you know, and and but it didn't happen overnight either, right?
• 38:02
  SPEAKER_04: I mean, a lot of companies.
• 38:04
  SPEAKER_01: No one's gonna use a credit card to buy something on the internet.
• 38:07
  SPEAKER_01: It's crazy, yeah.
• 38:08
  SPEAKER_04: Well, exactly, you know.
• 38:09
  SPEAKER_04: So, I mean, all these things, I think it's just a cycle, you know, you know, I think enterprises are of course way.
• 38:16
  SPEAKER_04: Oh, so I'll tell you another funny thing.
• 38:18
  SPEAKER_04: Um, we were doing a call with uh like a media company, and um, we were had a team of engineers, maybe like 10 engineers on the phone, and then the the the manager of the team was on the zoom too.
• 38:31
  SPEAKER_04: And you know, he was like, Well, I don't know if we can use these tools, like security is gonna like you know, have a lot of issues, questions, blah, blah, blah.
• 38:38
  SPEAKER_04: And then one of the engineers, she goes, she goes, Isn't it funny that like security is gonna stop us from using these tools?
• 38:45
  SPEAKER_04: But everybody has Chat GPT up in a browser window, they're just using it like all day long, you know.
• 38:52
  SPEAKER_04: And it was sort of like it's kind of like that whole bring your own, you know, tech to work kind of thing.
• 38:56
  SPEAKER_04: It's like that all of the employees are like, I need to get this task done.
• 39:01
  SPEAKER_04: This is pretty useful tool.
• 39:02
  SPEAKER_04: I'm gonna use it, you know, like until you like block it or something, I don't care.
• 39:06
  SPEAKER_04: You know, it's shadow IT, shadow IT, exactly.
• 39:09
  SPEAKER_04: Shadow IT.
• 39:10
  SPEAKER_00: Well, that that shadow IT is funny though, because then people start bringing in um things like, Well, okay, um, then I'm gonna do it on my personal iPad and I'm gonna pull it up and then I'm gonna send it across text to my laptop, and then I'm gonna copy and paste.
• 39:25
  SPEAKER_00: So, yeah, talk about the shadow of shadow IT.
• 39:28
  SPEAKER_04: Yeah, yeah, totally right.
• 39:30
  SPEAKER_01: So it kind of begs the question of um governance and decision making, right?
• 39:34
  SPEAKER_01: And governance is usually uh a bad word in a lot of organizations, like you know, uh when when can you meet to talk about governance?
• 39:42
  SPEAKER_01: And the response is uh, how about never?
• 39:45
  SPEAKER_01: Never never works for me.
• 39:47
  SPEAKER_01: Does never work for you?
• 39:49
  SPEAKER_04: And um I was gonna say when you said governance, I was like a small part of me inside died a little settled.
• 39:54
  SPEAKER_04: Yeah, yeah.
• 39:55
  SPEAKER_04: We're gonna talk about governance now.
• 39:56
  SPEAKER_01: Yeah, exactly.
• 39:58
  SPEAKER_01: So I like to call it metrics-driven decision making, right?
• 40:01
  SPEAKER_01: Yeah, yeah.
• 40:02
  SPEAKER_01: How do you make decisions and how do you measure them?
• 40:04
  SPEAKER_01: So, where do you see that kind of evolving?
• 40:06
  SPEAKER_01: Who on says, I mean, it's seems like a lot of people have to be brought to the table, and um, there's a lot of different factors here, right?
• 40:14
  SPEAKER_01: And but you know, governance and decision making has always been a challenge with organizations.
• 40:19
  SPEAKER_01: You know, you stand things up, you get them deployed, and then you know, there's no long-term uh mechanism for ownership and evolution and continuous improvement in many cases, or it gets folded into a larger decision-making structure.
• 40:37
  SPEAKER_01: I mean, governance is just a decision-making body and decision-making you know, procedures, right?
• 40:42
  SPEAKER_01: Uh and and so the question becomes how does this get folded in?
• 40:46
  SPEAKER_01: Who should own it?
• 40:48
  SPEAKER_01: Have you seen that um those kinds of structures come into place?
• 40:51
  SPEAKER_01: And and you know, what what are organizations gravitating toward when it comes to longer-term decision making and funding and you know, measurement of benefits versus risks.
• 41:05
  SPEAKER_04: Yeah.
• 41:05
  SPEAKER_04: Well, I think I think there's a lot of models out there right now, just because it's it's so new, right?
• 41:12
  SPEAKER_04: And there's a little bit of like this gold rush mentality of like, you know, this could be super big for our business.
• 41:18
  SPEAKER_04: So let's try this out, or we should be experimenting.
• 41:20
  SPEAKER_04: And so, so I think I, you know, seen a bunch of different models.
• 41:23
  SPEAKER_04: I mean, one, we've kind of seen people talk about like, oh, we have an AI council.
• 41:27
  SPEAKER_04: Um, and so we have like a group of people um from around the business and you know, IT development, whatever, that are kind of helping us evaluate these things, think about, you know, what are the big use cases, what are the big opportunities for our business, what is our evaluation process, you know, from a deployment, costs, you know, security, you know, data management, all those, all those kinds of things.
• 41:54
  SPEAKER_04: So that's kind of one thing we we've seen.
• 41:57
  SPEAKER_04: I think another model we've seen is where, you know, like you have like a security team that typically is evaluating, you know, that runs a questionnaire to understand like how do you handle data, you know, what are the, you know, um uh what you know, do you have SOC2 compliance?
• 42:17
  SPEAKER_04: You know, like what happens with our code, you know, like all the things that they normally use to evaluate a new tool.
• 42:22
  SPEAKER_04: So it's kind of been thrown onto them, like, hey, you should also now, you know, add in whatever you need to understand from an like AI perspective and what and what what questions you have.
• 42:32
  SPEAKER_04: And that sometimes is going to security and sometimes going to legal.
• 42:35
  SPEAKER_04: Um, and so those are kind of the probably the three biggest, you know, approaches we've seen so far.
• 42:44
  SPEAKER_04: Um I think some of this is because it's a little bit new, and so you know, that people like I think when there's an AI council, they tend to be a little bit more organized about it because they're just you know, it's a group that's been tasked with figuring this out.
• 42:58
  SPEAKER_04: I think when it's when it's thrown over the wall the legal or security, they're they're scrambling a little bit more just because they're like, well, who really owns this here and who's our expert?
• 43:07
  SPEAKER_04: But you know, I would say that teams are skilling up.
• 43:11
  SPEAKER_04: My my guess is that you know it'll take some time, but let's say by the end of next year, I think you know it'll be a pretty um it'll be a more uh well understood, you know, new technology.
• 43:25
  SPEAKER_01: Yeah, yeah.
• 43:27
  SPEAKER_01: So when you think about um kind of the uh the future and and your organization, uh, you know, where do you see the biggest opportunities and what are the biggest risks?
• 43:40
  SPEAKER_01: Where do you see, you know, what what are the types of problems that that you're very clearly solving today?
• 43:46
  SPEAKER_01: And and then what are the risks for you um down the road?
• 43:51
  SPEAKER_04: Yeah, yeah.
• 43:53
  SPEAKER_04: So I mean, I think you know, we really envision a world where there's gonna be a huge you know, unlock in the productivity capability.
• 44:00
  SPEAKER_04: For software developers.
• 44:01
  SPEAKER_04: You know, we think over the next five to seven years, you know, they're gonna see a five to 10X improvement in capabilities, right?
• 44:09
  SPEAKER_04: In terms of their ability to create code, have it be more secure, all the things we've been, you know, sort of talking about.
• 44:15
  SPEAKER_04: And so I think that's a huge, you know, set of capabilities that are gonna that are come to help software development, you know, um improve.
• 44:26
  SPEAKER_04: And so I think a lot of the things that we're doing today are kind of more like toys, frankly, you know, like like you were mentioning, Chris, like you know, line completions and things like that.
• 44:35
  SPEAKER_04: Um I mean, that's nice.
• 44:37
  SPEAKER_04: That's I feel like it's a little bit like excite search before Google came along, you know.
• 44:42
  SPEAKER_04: Um like it's cool, it's not gonna like 100 X you or something, you know.
• 44:47
  SPEAKER_04: Um, and so it did and so so I so I think that, you know, we think that like when we look at like the North Star, we're like, wow, there's a lot of you know possibilities here.
• 44:57
  SPEAKER_04: Um and and we think software developers are you know gonna go at a really rapid pace.
• 45:03
  SPEAKER_04: I think one of the risks, you know, I mean I wouldn't say it's like a I think it's a risk of a it slows down adoption is like one of the things we're noticing is like human like the adoption curve, you know, like the behavior change is required from people, you know.
• 45:20
  SPEAKER_04: And uh it's interesting we see, like, I mean, this is probably you know typical of like many technology shifts, like younger people are more, you know, adaptive to these things, and people who are more set in their ways or been doing something for a long time a certain way are like you know, less interested trying things out, you know, and less they're just like, Oh, I have a process that works, I don't really want to try something new, you know.
• 45:46
  SPEAKER_04: Um, just to give you like a like an example.
• 45:50
  SPEAKER_04: So, like my nine-year-old, um, and it's just funny because I watch him because I see him work with these tools, and I'm like, oh, this is gonna be how people probably do it.
• 46:00
  SPEAKER_04: So he doesn't really know how to write code, right?
• 46:03
  SPEAKER_04: I mean, he's kind of learning, and I'm trying to get him to do some JavaScript tutorials and Python tutorials, but like he doesn't really know.
• 46:09
  SPEAKER_04: But he just basically like grabs some code from somewhere and then try, puts it in his IDE and then tries to run it, then he gets a build error, and then he just throws it into the LLM and he's like, here's my error, fix it, rewrite the code, you know, and then it like rewrites the code again, and then he tries to build it and he's like, No, you make this change, rerun it, fix it, right?
• 46:29
  SPEAKER_04: So he just takes this iterative loop.
• 46:31
  SPEAKER_04: I mean, I was like, What?
• 46:32
  SPEAKER_04: You can't write software like that, you know.
• 46:34
  SPEAKER_04: We gotta like look at each line and figure out what's the problem, and you know, and so it's like that's because I'm old.
• 46:41
  SPEAKER_04: So, you know, and so like it's a behavior change for me, you know.
• 46:45
  SPEAKER_04: Um, I mean, think about when you have like a million context, right?
• 46:48
  SPEAKER_04: You know, tokens you can give the model, and it's a we're on GPT 7.
• 46:53
  SPEAKER_04: I mean, it's just gonna be a totally different, you know, set of capabilities.
• 46:56
  SPEAKER_04: And so I think that, you know, we're trying to like make these tools work how people work today, and trying to look at the future.
• 47:05
  SPEAKER_04: But I think, I think as it takes time for people to have some people love, like we have a chat product, right?
• 47:10
  SPEAKER_04: So it's like you can, it's a chat window in your IDE and you can use it to ask questions, have a conversation, you know, blah, blah, blah, blah.
• 47:17
  SPEAKER_04: And a lot of people like love it.
• 47:19
  SPEAKER_04: And then other people are like, oh, I don't, I'm not into chat, you know.
• 47:23
  SPEAKER_04: And so I'm like, yeah, but this could really help you go a lot faster.
• 47:28
  SPEAKER_04: And they're just like, Yeah, I'd like to just write my code in my, you know, IDE, you know, uh, in my text editor.
• 47:33
  SPEAKER_04: And so I'm not saying anything's wrong with those people.
• 47:35
  SPEAKER_04: I'm just saying that like it's a behavior change, and some people want to make it and some people don't, you know.
• 47:41
  SPEAKER_00: Well, that was that was like uh there was, you know, many moons ago I was at Microsoft, and I remember us working with the developer commit developer community, and we looked, went and saw this guy's code, and it was all tightly bound to like this very, very thin column.
• 47:56
  SPEAKER_00: And I was like, what in the hell is wrong with uh you know his code?
• 48:00
  SPEAKER_00: Why isn't it, you know, like just expanded out and stuff?
• 48:03
  SPEAKER_00: It's because he was copying, you know, code from the MSDN magazine column.
• 48:09
  SPEAKER_00: And so he thought that his code had could only be this long, right?
• 48:12
  SPEAKER_00: And so all of his code was that long, and I was like, oh my gosh, think about the behavioral trends of of following somebody else, right?
• 48:20
  SPEAKER_04: Yeah, yeah.
• 48:21
  SPEAKER_00: And it's those kind of behaviors talking, you know.
• 48:23
  SPEAKER_00: I mean, probably dating myself now before um, you know, before you know those kind of runtimes into was it on the Gutenberg press?
• 48:30
  SPEAKER_00: Yeah, exactly.
• 48:32
  SPEAKER_00: Exactly, exactly right.
• 48:33
  SPEAKER_00: Yeah, with um, with you know, and if you can imagine if you missed the semicolon on that, right?
• 48:39
  SPEAKER_00: Anyway, exactly.
• 48:40
  SPEAKER_00: But uh, but it was just funny because it's that that same those same ideas still apply around when folks are are trying to copy other people's code and put it in there, just because it's there doesn't mean it's good, but that's where I think a lot of this really great tools will come in to say, hey, that's one way to do it, but just so you know, it's not optimized.
• 49:02
  SPEAKER_00: Let me help you re-optimize that if you're gonna do copy and paste.
• 49:06
  SPEAKER_00: Because I mean, hell, we all copy and paste, but you know, pull all these things together.
• 49:10
  SPEAKER_00: But yeah, really, really fun problems to solve.
• 49:13
  SPEAKER_00: So my hat's off to you.
• 49:14
  SPEAKER_01: So we're almost at the top of the hour, and uh, I did want to ask you a question.
• 49:19
  SPEAKER_01: So uh, you know, where you where where are you located and what do you do for fun?
• 49:24
  SPEAKER_01: And then I have one other uh quick question for you.
• 49:28
  SPEAKER_04: Yeah, um, I'm located, I'm in the Bay Area, I'm in Menlo Park, California, right next to Stanford.
• 49:34
  SPEAKER_04: Um, so yeah, um Is that where you're yeah yeah?
• 49:36
  SPEAKER_04: I was actually born uh I was born here in Mountain View, and then I I've lived it was in New York for a long time.
• 49:42
  SPEAKER_04: I lived in India for about 10 years, but then I um my me and my family we moved back here uh about six, seven years ago.
• 49:49
  SPEAKER_04: Yeah, so we've been back here for a while.
• 49:50
  SPEAKER_04: Nice to be home, actually.
• 49:54
  SPEAKER_04: Yeah, what'd you do for fun?
• 49:56
  SPEAKER_04: What I do for fun, to be honest, right now I'm working pretty hard on Pito, but um uh I like to play you wonder why you have no sp no spare time.
• 50:06
  SPEAKER_04: Yeah, yeah.
• 50:07
  SPEAKER_04: No, I like to play golf, I don't play a lot, but I I enjoy it.
• 50:11
  SPEAKER_04: And um I I feel like I I'm I'm getting better the less I play, which is a little bit weird.
• 50:18
  SPEAKER_04: Um, maybe that says something about my golf game.
• 50:21
  SPEAKER_04: Um, so yeah, like to like to like to do that.
• 50:24
  SPEAKER_04: And you know, my kids are at a fun age right now, so just you know, went camping with them last weekend, just you know, doing all that stuff before they don't want to spend time with mom and dad anymore.
• 50:32
  SPEAKER_04: So yeah, how old comes?
• 50:33
  SPEAKER_01: How old are the kids?
• 50:35
  SPEAKER_04: They're nine and eleven.
• 50:36
  SPEAKER_04: So two boys, yeah.
• 50:38
  SPEAKER_04: So it's kind of a you know fun age, I think.
• 50:39
  SPEAKER_00: So they're they're still they're still in their puppy dog face, so they still love mom and dad, and then they'll turn into cats, they'll turn into cats, come on, and then not they won't come back to like their you know you know mid-20s.
• 50:51
  SPEAKER_04: Yeah, yeah, I know we're we're kind of getting that, but hearing that from friends who are like a little ahead of us, you know.
• 50:57
  SPEAKER_01: So one last uh introspective question if you go back um in time and the time you graduated from college and you could give yourself a piece of advice, what might what might that be?
• 51:11
  SPEAKER_04: Yeah, that's a good question.
• 51:13
  SPEAKER_04: Um you know, I think um I think as I've gotten like older, you know, I've definitely thought about that.
• 51:23
  SPEAKER_04: And I and I guess I would say that, you know, I think uh it's a little bit like uh, you know, kind of Buddhist philosophy or Hindu philosophy.
• 51:32
  SPEAKER_04: I mean, just kind of lower your expectations, you know, in the sense of like, I think when you when you kind of approach a situation with it has to be this way and then it's not, that's what creates anger and frustration and tension, you know, and all that.
• 51:49
  SPEAKER_04: And so, you know, just kind of accept the world and people for you know who they are and deal with the situations as they come, you know.
• 51:58
  SPEAKER_01: So that's no something no expectations, right?
• 52:02
  SPEAKER_04: Yeah, yeah.
• 52:03
  SPEAKER_04: So I mean it's not to say that you have a low bar, it's not to try to say you're like fine with whatever happens, but you know, it's it's about trying to like uh accept the situation for what it is versus what you wanted it to be and then deal with it from there, right?
• 52:17
  SPEAKER_01: No, that's a great, that's a great uh uh great philosophy.
• 52:21
  SPEAKER_00: And what I try to try to uh abide by you know um I just I just try to tell it, say listen, if you're flexible, you won't get bent out of shape, right?
• 52:29
  SPEAKER_00: And that's the there you go.
• 52:32
  SPEAKER_01: Um it's been great to have you.
• 52:34
  SPEAKER_01: Really thank you.
• 52:35
  SPEAKER_01: Thank you so much for your time today.
• 52:37
  SPEAKER_04: Yeah, thank you.
• 52:38
  SPEAKER_04: Really, really great to chat with you guys, and thank you so much.
• 52:40
  SPEAKER_04: And it was fun.
• 52:41
  SPEAKER_04: And uh yeah, look forward to the feedback from your listeners.
• 52:43
  SPEAKER_01: Absolutely, and thank you to our audience, and uh, we will see you next time.
• 52:49
  SPEAKER_01: This has been another uh episode of the Early AI podcast, and we'll see you soon.
• 52:56
  SPEAKER_03: Thanks, Mark.
• 52:57
  SPEAKER_03: Thank you, thanks, guys.
• 52:59
  SPEAKER_03: Appreciate it.
• 53:01
  SPEAKER_03: Thank you for joining us on another deep dive into AI innovation.
• 53:05
  SPEAKER_03: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 53:11
  SPEAKER_03: The Early AI Podcast is sponsored by Early Information Science and CMS Wire.
• 53:16
  SPEAKER_03: To learn more, visit early.com.
• 53:19
  SPEAKER_03: That is E A R L E Y dot com.
• 53:23
  SPEAKER_03: Thanks for tuning in.