Latest / The Adults In The AI Room / Earley AI Podcast Episode 67: How AI Is Transforming Software Engineering With Yang Li of Cosine

Transcript

Earley AI Podcast Episode 67: How AI Is Transforming Software Engineering With Yang Li of Cosine · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Zephyr.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as Vector.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_01: Well, welcome to today's Early AI Podcast.
• 0:32
  SPEAKER_01: I'm Seth Early, your host, and today we're diving into how AI is changing the landscape of software development.
• 0:38
  SPEAKER_01: With AI models advancing rapidly, the role of developers is shifting in big ways.
• 0:42
  SPEAKER_01: And we'll break down what that means for the industry and what's really next.
• 0:46
  SPEAKER_01: Our guest today is Yang Lee.
• 0:48
  SPEAKER_01: He's a leader in AI and software innovation.
• 0:50
  SPEAKER_01: Yang has a strong track record in scaling startups and navigating complexities of AI technology.
• 0:56
  SPEAKER_01: Currently, he's chief operating officer at cutting-edge AI development firm Cosine.
• 1:01
  SPEAKER_01: He's played a key role in everything from pioneering a micromobility IPO to driving major advancements in engineering software.
• 1:09
  SPEAKER_01: His work focuses on leveraging AI to empower the next generation of developers.
• 1:13
  SPEAKER_01: Yang, welcome to the show.
• 1:15
  SPEAKER_01: Thanks for having me, Seth.
• 1:16
  SPEAKER_01: Let's start off with the big picture question.
• 1:19
  SPEAKER_01: When you start thinking about AI and development, how is AI reshaping the role of software engineers today?
• 1:25
  SPEAKER_01: We know that there's lots of different approaches for doing code generation.
• 1:29
  SPEAKER_01: We know there's GitHub copilots and other types of uh tools.
• 1:33
  SPEAKER_01: With AI shifting from hands-on coding to more strategic and architecture-driven approach, what do you see as the key elements of that transformation?
• 1:43
  SPEAKER_02: Yeah, I think that, you know, my view is that it's become quite a binary environment, right?
• 1:48
  SPEAKER_02: So on the one hand, which I think which is very publicly noticeable, developers are pushed towards becoming much more creative, to build experiment much more, iterations much faster, build whether it's new, brand new apps or software or even brand new features, this vibe coding phenomenon has pushed developers to become much more creative.
• 2:11
  SPEAKER_02: They become the ideas person.
• 2:13
  SPEAKER_02: So they have every skill set and no excuse now to not become the pioneers of new ideas.
• 2:18
  SPEAKER_02: On the other side, I think what is less talked about is also the vast quantities of backlog tasks, right?
• 2:25
  SPEAKER_02: You know, any organization that has software more than a couple of years old with more than a million or two lines has a huge backlog of whether it's bugs that are non-critical, features that you know are important to some customers, but probably not enough to divert the entire engineering team towards, or even just nice to have, like these ideas that are more rounded but just require more engineering effort to do, but are sitting on a backlog.
• 2:49
  SPEAKER_02: I think that is now the next frontier where people are focusing on to say it's fun and it's useful to have greenfield projects and vibe to put your way through that.
• 2:59
  SPEAKER_02: But what do we do now in terms of the backlog that we've had maybe years, if not decades, of?
• 3:04
  SPEAKER_02: Yeah.
• 3:05
  SPEAKER_01: So so that's that's really super important.
• 3:07
  SPEAKER_01: And I know there's some organizations, there was an insurance company we worked with years ago that had, you know, of course, they still have COBOL, right?
• 3:14
  SPEAKER_01: And they had so many systems and processes and just such complexities that trying to innovate was really a nightmare for them.
• 3:21
  SPEAKER_01: And that that legacy code base that that they had was just insurmountable in terms of trying to modernize that.
• 3:28
  SPEAKER_01: So I imagine that that's certainly going to be true for some organizations.
• 3:34
  SPEAKER_01: How do you how do you really see that being addressed in in this uh AI-driven software engineering environment and and um landscape?
• 3:43
  SPEAKER_02: Yeah, I think it's it's a question or a problem that we were posed quite early on.
• 3:49
  SPEAKER_02: When we started Cosign, you know, we of course were dough-eyed and idealistic and thought, you know, in a few weeks we could create a solution that every investment bank and insurance company under the sun would use.
• 4:00
  SPEAKER_02: And you know, quite frankly, we were put it in our place very early on.
• 4:05
  SPEAKER_02: Another from our side is that we spoke to a you know, a bold bracket investment bank that flew us out, present we presented to the managing directors, and they just asked us point blank how are you going to help us make PRs uh, you know, and ship new code in COBOL, in Haskell, in Assembly, you know, to a certain extent that a lot of these languages are no longer taught.
• 4:28
  SPEAKER_02: Like no, no college student is learning uh you know COBOL or Fortran anymore.
• 4:32
  SPEAKER_02: And so what do we do?
• 4:34
  SPEAKER_01: And there's not a not a lot of training examples out there, I would imagine.
• 4:37
  SPEAKER_02: Exactly, right?
• 4:37
  SPEAKER_02: And and even if there are training examples, then they're not open source.
• 4:40
  SPEAKER_02: They're behind their private repos inside these enterprises that they wouldn't never let you have access to.
• 4:45
  SPEAKER_02: And so that's that's fundamentally a lot of these important and you know out-of-distribution sort of examples are not also not trained inside LLMs, no matter what model you use, whether you train your own, whether you use an open source or a closed source, open AI, anthropic, a meta model.
• 5:01
  SPEAKER_02: It's just out of distribution.
• 5:03
  SPEAKER_02: So what can you do about it, right?
• 5:05
  SPEAKER_02: So we took the perspective of you only have two options in in our view, right?
• 5:10
  SPEAKER_02: You have some players in the space who are doing agentics software engineers like we are.
• 5:15
  SPEAKER_02: You you kind of brute force it.
• 5:16
  SPEAKER_02: You rely on inference-time compute, you throw as many solutions as possibly for humanly or even a GPU possibly that you can generate.
• 5:25
  SPEAKER_02: So you're somewhat throwing spaghetti at the wall, hoping that one of these solutions will work.
• 5:29
  SPEAKER_02: And in an infinite time frame with enough compute, I think it is possible, but it's probably not very probable that you'll land on something that adheres to you know four-trand code that was written in 1971.
• 5:41
  SPEAKER_02: I think that's the problem, and that's what we always wanted to focus on as an organization.
• 5:45
  SPEAKER_02: We knew that there were lots of players and competitive space in a Greenfield project, in a modern code base, whether it's TypeScript or JavaScript.
• 5:53
  SPEAKER_02: But what we wanted to do was, well, how do you train AI to be good at legacy code bases, you know, uh coding languages that have fewer examples?
• 6:03
  SPEAKER_02: Well, we fundamentally said, well, how does a human do so, right?
• 6:07
  SPEAKER_02: So we thought, well, if you are a human, if you were dumped inside a code base you were unfamiliar with, and if you're looking at a language that you haven't learned fully, you would rely on two things.
• 6:17
  SPEAKER_02: One is that you would kind of browse, right?
• 6:20
  SPEAKER_02: You would gather information, you would open some files, you would read documentation, you would, you know, to some extent, you would look at comments that have been left by other developers in the past.
• 6:29
  SPEAKER_02: And on the other side, you would rely upon your other knowledge to generalize across, right?
• 6:34
  SPEAKER_02: So you would look at other languages that you would be familiar with to do so.
• 6:38
  SPEAKER_02: And so that's what we did.
• 6:39
  SPEAKER_02: We uniquely have uh you know, generated a synthetic data set.
• 6:43
  SPEAKER_02: And this is from day one we've always said that mimics how a human would reason and navigate a code base and do you know coding tasks, right?
• 6:51
  SPEAKER_02: So what we did is that we essentially uh are able to generate uh that data set that teaches or shows the AI how exactly for types of tasks, what files you would open, what what are the steps you would take next, so that there is some sort of semblance.
• 7:07
  SPEAKER_02: And I'm not claiming that we've instilled reasoning inside AI model, but it's it's some sort of mimicry of reasoning.
• 7:13
  SPEAKER_02: And so what that allows us to do is that our product and our AI model that powers it is able to take an unfamiliar code base, an unfamiliar task, and kind of work through it and generate much more signal, right?
• 7:25
  SPEAKER_02: So that it can become much better at doing these tasks.
• 7:29
  SPEAKER_01: So so that's interesting.
• 7:30
  SPEAKER_01: And and how well has that worked in practice?
• 7:34
  SPEAKER_01: Have you been able to tackle some um revamping and and uh uh redeployment of uh of that type of legacy um software?
• 7:44
  SPEAKER_02: Yeah, so I I think that there is two data points uh to point out here.
• 7:48
  SPEAKER_02: One is, of course, everyone in this industry cares about benchmarks, and our latest benchmark is we ran Swede Lancer, which is an open AI release coding agent benchmark, which is they took a million dollars worth of real developer tasks on upwork.
• 8:02
  SPEAKER_02: So these are real bounties that uh someone was willing to pay for and then completed by a freelancer, and then they pit essentially AI coding agents up across those million dollars of tasks.
• 8:13
  SPEAKER_02: We scored, you know, we essentially scored 49% uh across the benchmark, and and that is significantly higher.
• 8:21
  SPEAKER_02: And don't quote me on this, please don't check this.
• 8:23
  SPEAKER_02: My believe is like uh so if you look at OpenAI's GPT-4, they were at 8%, and uh 01 was at 16%, right?
• 8:32
  SPEAKER_02: So it is a significant jump in terms of its ability to solve problems of real developer tasks, which we hope, right?
• 8:38
  SPEAKER_02: And and no benchmark is perfect, it's always room from benchmarks, but it is a cross-section of a real developer task across the spectrum.
• 8:45
  SPEAKER_02: I was saying, like, you know, and I think even you know, the the second data point is that you know we we now do have uh since we last like connected, we we we do have a publicly available product now everyone can access.
• 8:56
  SPEAKER_02: And one of one of the larger enterprises that we deployed at is a is a large um you know, you know, bulge bracket investment bank that that is specifically asking us to tackle the the problems that their engineers cannot do, right?
• 9:09
  SPEAKER_02: And um, you know, I was I can't I can't tell promise you what the long-term aspect is because we've just launched a week ago, but early on it's it's very promising in terms of like, you know, just even if you are super skeptical and say, you know, I don't believe in a gentic software engineers yet.
• 9:23
  SPEAKER_02: We have seen that time to get a human engineer up to speed on a language or uh or a repo that they are not familiar with has decreased dramatically.
• 9:33
  SPEAKER_02: Where we're talking orders of magnitude faster.
• 9:36
  SPEAKER_01: Interesting.
• 9:37
  SPEAKER_01: So so it's essentially helping the engineer be able to tackle these problems.
• 9:42
  SPEAKER_01: Now you said they were not able to tackle the problems.
• 9:44
  SPEAKER_01: What was the what was the big um obstacle when you said their engineers couldn't do some of these things?
• 9:49
  SPEAKER_01: Was it just lack of experience and fluency in the in the uh languages because they're so archaic?
• 9:54
  SPEAKER_01: Was it the level of complexity of the of the different systems and tools?
• 9:58
  SPEAKER_01: What was the um a big obstacle?
• 10:01
  SPEAKER_02: Yeah, I think that I can't say for certain, but at least from the data that I I have seen, the my my gut feel is that the vast majority is from bad prompts, right?
• 10:11
  SPEAKER_02: So there's two, there's essentially a couple of ways that you can use our product genie.
• 10:15
  SPEAKER_02: You can just type it into a chat box, give it a prompt, you can copy and paste a ticket from linear, you can do a direct integration, so you can assign the linear task or Jira task, or you can send it from Slack.
• 10:27
  SPEAKER_02: When the prompt or the ticket is detailed enough, then our agent is is much more capable.
• 10:33
  SPEAKER_02: It's usually the lack of information that is problematic, right?
• 10:36
  SPEAKER_02: So for a human, you can be much more um flexible in terms of being vague, and you know, if you said something ambiguous.
• 10:44
  SPEAKER_02: No, yeah, ambiguously say, you know, uh fix the fix the bugs on the front end.
• 10:48
  SPEAKER_02: That that that maybe because you've spoken to a colleague at a water cooler, you wouldn't understand that.
• 10:52
  SPEAKER_02: But for an agent, because you don't have that context, it just doesn't work.
• 10:55
  SPEAKER_02: So if you can overcome that problem, the next problem we have is, and I don't think this is unique to us, I think this is a lot of coding agents are the same, is that if the task is too large or too too big, right, it events essentially it kind of loses its way.
• 11:11
  SPEAKER_02: I'd say it's you know, not to humanize it too much, it almost gets distracted.
• 11:14
  SPEAKER_02: And so what we are about to release next week is a new feature called Auto PM.
• 11:18
  SPEAKER_02: So, you know, we as a it's an auto project manager.
• 11:21
  SPEAKER_02: We see it as our version of uh deep research as like an open AND where it can ask further questions, it essentially puts itself into a loop and it can go for hours, right?
• 11:32
  SPEAKER_02: We've given it tasks where it has come back three hours later and successfully deployed, you know, a huge change, right?
• 11:38
  SPEAKER_02: Where there is like tens of thousands of lines of differences.
• 11:41
  SPEAKER_02: And I think for us that that's the next step, what we're trying to fix.
• 11:44
  SPEAKER_02: There's always a limit of what we we think that, as you know, as you ship, as you think you fix somewhere, there's always a new leak that appears somewhere else.
• 11:51
  SPEAKER_02: But I think we are moving towards both user education becoming better, but also a product becoming better much more truly agentic, right?
• 11:59
  SPEAKER_02: I saw something in X the other day.
• 12:01
  SPEAKER_02: It's like you can't really claim your uh AI tool as agentic unless it basically when you have to prompt it multiple times for the same task.
• 12:08
  SPEAKER_02: And I think we we probably agree, right?
• 12:10
  SPEAKER_02: When we hopefully our next update will decrease that, but we're not truly agentic fully until we get to the point where it's completely autonomous by itself.
• 12:18
  SPEAKER_01: Well, I I guess I would not completely agree with that.
• 12:21
  SPEAKER_01: I would think that you still have to give course corrections, you still have to give ongoing instructions and prompts and so on.
• 12:28
  SPEAKER_01: But I understand the point.
• 12:29
  SPEAKER_01: Yeah.
• 12:30
  SPEAKER_01: Um, I'm not sure I would agree with it completely.
• 12:32
  SPEAKER_01: I think you could still be agentic and not and still have some interventions.
• 12:35
  SPEAKER_01: Anyway, I don't want to argue that point.
• 12:37
  SPEAKER_01: But but my question, uh let me rephrase my question a little bit, because when you first mentioned this, you said this uh large uh investment bank was asking you to solve problems that their engineers could not solve.
• 12:50
  SPEAKER_01: And so can you talk more about what was holding the engineers back?
• 12:53
  SPEAKER_01: So where how did you fill that gap?
• 12:55
  SPEAKER_01: What was the specific capability that he did not have, or what was the reason?
• 13:00
  SPEAKER_01: Was it simply a matter of bandwidth?
• 13:02
  SPEAKER_01: Was it simply skill set?
• 13:03
  SPEAKER_01: Was it complexity?
• 13:04
  SPEAKER_01: What were the things that that were stopping the agent, the engineers from being able to solve those problems?
• 13:09
  SPEAKER_02: Yeah, I think there's a couple of different categories, but you're absolutely right with the first one.
• 13:13
  SPEAKER_02: One is number one is just bandwidth.
• 13:14
  SPEAKER_02: A lot of these times the engineers do not have the capacity to get to it.
• 13:18
  SPEAKER_02: Like when we talked about the backlog, these backlogs sometimes decade long, if not more.
• 13:23
  SPEAKER_02: What they simply do is that they assign every task to our agent.
• 13:27
  SPEAKER_02: And speaking very candidly, some tasks we only return a result that is 20, 25% useful.
• 13:33
  SPEAKER_02: Other times we return something that is 99% useful.
• 13:36
  SPEAKER_02: But it is still better than zero.
• 13:38
  SPEAKER_02: And then what they do is that again, the feedback we've gotten is even if it doesn't completely get it right, it's much easier for an engineer to critique the work than it is for them to start on a blank sheet of paper and have no context.
• 13:49
  SPEAKER_02: And so they they are able to work asynchronously, and even to the extent where they still do not assign any engineers fully to this backlog.
• 13:59
  SPEAKER_02: What they do is to say, well, the engine, the human engineers, you're assigned for this print, you still have your own tickets that you have to prioritize.
• 14:06
  SPEAKER_02: But in your downtime or asynchronously, take a preview or take a peek at what a genie has returned.
• 14:12
  SPEAKER_02: And if it takes you know one or two prompts, or it takes, and we we essentially allow the developers to have the same um you know code editor that Genie has access to.
• 14:21
  SPEAKER_02: So you could just literally dive in and change a few lines of code.
• 14:24
  SPEAKER_02: If that is all it takes to ship the PR, and what Genie does is it runs your your CI fully, right?
• 14:30
  SPEAKER_02: So it you have some level of confidence already that the PR its shipping will work if it passes your CI.
• 14:35
  SPEAKER_02: It allows them to basically juggle and multitask much more effectively.
• 14:38
  SPEAKER_02: So that's the first category.
• 14:39
  SPEAKER_02: The second category is I also see a lot of problems, particularly with legacy code bases, which is simply size of code base.
• 14:46
  SPEAKER_02: No engineer can memorize a million lines of code and you know how it works and interoperates and all that stuff.
• 14:52
  SPEAKER_02: And it's simply, you know, whether it's a new engineer or an experienced engineer, it's it's allowing them to essentially say, okay, I now have some idea where to go.
• 15:00
  SPEAKER_02: And even if the agent is completely wrong, it started my thinking in towards that direction.
• 15:05
  SPEAKER_02: And it it it's it's this essentially to solve the cold start problem, right?
• 15:10
  SPEAKER_02: I think non-engineers or people who haven't done software engineering before view it as you know, the engineer sits down, immediately starts writing code as if it's in a Hollywood movie, and then at the end of the day, they ship something and it works magically.
• 15:23
  SPEAKER_02: They're probably, I think most engineers would agree, the vast majority of the time is thinking and researching and navigating.
• 15:29
  SPEAKER_02: And and that for what we what we do is that we eliminate that cold start problem.
• 15:33
  SPEAKER_02: Um, and sometimes 100% of the time, and sometimes 20% of the time, right?
• 15:37
  SPEAKER_02: And then it that doesn't matter.
• 15:39
  SPEAKER_02: And the very last category, I think, for us that is um surprising, right?
• 15:44
  SPEAKER_02: And and maybe this is not fully surprising, not surprising inside an IDE.
• 15:48
  SPEAKER_02: I think that it's become the norm for people to use uh AI-enabled ID like cursor or windsurf, and it's it's but that's on device, right?
• 15:56
  SPEAKER_02: So someone you can't really do asynchronously, and you know, if you wanted to a lot of times if you wanted to access your terminal and do commands, you have to be present to allow it.
• 16:06
  SPEAKER_02: Asynchronously on an agent, we have found that it's uh it's almost a creativity problem that we're solving.
• 16:14
  SPEAKER_02: So when an engineer is given a vague uh task to say improve uh conversion or improve loading speed, right?
• 16:23
  SPEAKER_02: We have seen people do the same task concurrently in multi in multiple instances.
• 16:28
  SPEAKER_02: So the same ticket is prompted slightly differently 20 different times, and then they the engineer then comes back, you know, let's say 30 minutes later, and then they see which one has promise.
• 16:38
  SPEAKER_02: So it's it's A-B testing on a massive scale where they they couldn't think of every single idea, they then tested every single idea, and then they essentially eliminate the ones that are not working very aggressively.
• 16:50
  SPEAKER_02: And that is something that we've seen a lot of the larger organizations that we do, because obviously the larger organizations have lots of voices and lots of great ideas, but how do you filter down to the right idea?
• 17:01
  SPEAKER_02: Well, data or performance speaks for itself, right?
• 17:03
  SPEAKER_02: So why not try it?
• 17:05
  SPEAKER_02: And to enable people to do it asynchronously, multiple instances at the same time, concurrently, is something that is only achievable if you have an agent and not inside your IDE, right?
• 17:15
  SPEAKER_02: And and also for us, you know, the way that our pricing works, right?
• 17:20
  SPEAKER_02: And you know, people can go on our website and look for pricing on cells, so we'll promote ourselves here, but it's flat, right?
• 17:24
  SPEAKER_02: So I I think for us it's like people are now increasingly worried about using AI-enabled IDEs and the price that yes, there is a fixed monthly subscription, but then the number of tokens and or you know computing units that you have is is suddenly going to skyrocket if you really rely on it.
• 17:40
  SPEAKER_02: And efficiency increases, but so does your cost.
• 17:42
  SPEAKER_02: I think that it's not as linear as people think, which is efficiency increases and then cost decreases.
• 17:48
  SPEAKER_02: A lot of organizations have not seen their cost decrease.
• 17:51
  SPEAKER_02: They've actually seen their cost increase despite the productivity increase that they're having.
• 17:55
  SPEAKER_01: Right.
• 17:56
  SPEAKER_01: I mean, it's almost I almost think that that generative AI slows me down, right?
• 18:00
  SPEAKER_01: Because I will start with my materials.
• 18:02
  SPEAKER_01: I never ask it questions without context.
• 18:05
  SPEAKER_01: It has all of my content, it has articles, it has my tone and voice and all of that.
• 18:09
  SPEAKER_01: But then when I start processing my ideas, I have too many options, right?
• 18:15
  SPEAKER_01: I have to and I have to go through different iterations.
• 18:17
  SPEAKER_01: And so I agree, you know, the more possibilities there are, the more bandwidth it requires to evaluate those.
• 18:23
  SPEAKER_01: So hopefully that'll that'll improve.
• 18:24
  SPEAKER_01: Now, one of the things that we talked about in the earlier uh prep uh call was were benchmarks and how you mentioned at the time that you you're you did really well on a benchmark.
• 18:34
  SPEAKER_01: And maybe you could talk about that benchmark.
• 18:36
  SPEAKER_01: But then you said, but the product was still lacking from your perspective.
• 18:39
  SPEAKER_01: And you want to talk a little bit about benchmarks versus you know real-world product capabilities, and then what you found, you know, was what was that disparity?
• 18:49
  SPEAKER_02: Yeah, no, I I think you know, this was last summer when we we ran Sui Verified, right?
• 18:56
  SPEAKER_02: Which is an another coding agent benchmark that was industry standard.
• 19:00
  SPEAKER_02: At that time, we were the highest score in the world.
• 19:03
  SPEAKER_02: We haven't since run it.
• 19:05
  SPEAKER_02: And I think that it's quite indicative why we haven't rerun it because we didn't think it was particularly useful anymore to help us build a product.
• 19:11
  SPEAKER_02: But uh we still represent as the largest jump in score in its history, you know.
• 19:16
  SPEAKER_01: That's impressive.
• 19:17
  SPEAKER_02: Yeah, I'm arguing with that.
• 19:19
  SPEAKER_02: You can't argue with that data, right?
• 19:20
  SPEAKER_02: Yeah, we can't, and and I think and again, don't quote me on this.
• 19:23
  SPEAKER_02: I think we are also one of the few people, if not the only people in the world store that has run every single task on Sweetbench and Sweetbench verified.
• 19:31
  SPEAKER_02: It's just actually it's an incredible hard benchmark to get up and running.
• 19:34
  SPEAKER_02: Um and and and the problem is with with Sweetbench, is that at the time we blindly accepted because everyone else loved it, then if we beat it, then obviously our product will be great.
• 19:46
  SPEAKER_02: In hindsight, it was only Python, right?
• 19:50
  SPEAKER_02: And and then it was only about bug fixes.
• 19:52
  SPEAKER_02: Engineers across the world don't just use Python to fix bugs, they do such a wide variety of things.
• 19:57
  SPEAKER_02: And we we actually soft-launched that product to a small cohort of, you know, we had tens of thousands of people on our wait list, and we let a few a few like sort of select users and some smaller ones, some larger ones, on it to try it, but it just kept on not working, right?
• 20:12
  SPEAKER_02: And and then it was because it was it was such a disparity, was to say we had such a great benchmark result, but in the real world it wasn't working, and there was a couple of things that wasn't working.
• 20:21
  SPEAKER_02: Number one is that we the product itself just wasn't conducive in supporting how to work with an agentic uh AI coding agent, right?
• 20:33
  SPEAKER_02: At that time, our product was even if Genie got it 99.99% right, there was no way for you to make small changes with a code editor, and and you would have to reprompt it and hope for the best and hopefully it came out again.
• 20:45
  SPEAKER_02: And that was non-deterministic.
• 20:46
  SPEAKER_02: And and we were just too naive thinking that of course we it would it would work and you could just re-roll the dice, but that just wasn't the case.
• 20:54
  SPEAKER_02: And I think the second point that we we found was that you know, we uh actually found that it it's not just one benchmark.
• 21:05
  SPEAKER_02: You we actually had to create from the difference between last year and this year is that we now have a a plethora of internal benchmarks that are not public, right?
• 21:13
  SPEAKER_02: We we use the full candor here.
• 21:16
  SPEAKER_02: We do we do use the public benchmarks as some sort of PR and some to promote ourselves, and it's generally accepted, and people get very excited about it.
• 21:23
  SPEAKER_02: And it's it's it's part of the portfolio that we use today.
• 21:26
  SPEAKER_02: But a lot of the times now we we do is we have to create benchmarks in areas where there is no coverage on public benchmarks, right?
• 21:34
  SPEAKER_02: So, for example, of what about a code base, or what about a project that uses multiple languages at the same time, right?
• 21:41
  SPEAKER_02: Or what about you know using libraries that are severely outdated instead of just the most up to date libraries?
• 21:48
  SPEAKER_02: And that that to us is actually the biggest learning is that we've now have a dedicated team just to build in internal evals doing that.
• 21:56
  SPEAKER_02: And for us, then this is a I guess a little bit of I don't know.
• 22:00
  SPEAKER_02: My team behind me is going to kill me for saying this out loud, but we are, you know, we are we are very advanced in supervised fine-tuning, and this is what a current product uses.
• 22:09
  SPEAKER_02: But of course, like everyone else, and you know, there's an MIT review, tech review article about us in Poolside, both tackling the next frontier, which is you know, reinforcement learning with code execution, creating our own evals and reward functions is how you win the next frontier doing reinforcement learning sort of techniques.
• 22:25
  SPEAKER_02: And for us, we're already seeing for Genie3, our next model, huge leaps in sort of performance just simply by having really great internal evals that other people don't have.
• 22:37
  SPEAKER_02: And I do think that that's what's going to make or break our startup in the future about creating these coding agent models.
• 22:43
  SPEAKER_01: Interesting.
• 22:43
  SPEAKER_01: That's interesting.
• 22:44
  SPEAKER_01: So, so um one of the things that we had uh talked about is the danger of you know over reliance on some of these tools, just like we see you know, marketing organizations uh, you know, offload too much of their stuff to a gen AI application, and then their campaigns kind of sound like everybody else, and you know, they don't they don't have as much differentiation.
• 23:08
  SPEAKER_01: I like to say uh, you know, uh uh standardization when you're doing stuff that other people are doing will give you some efficiency, but differentiation gives you competitive advantage, right?
• 23:17
  SPEAKER_01: And now in the coding world, of course, you are there are always differentiating because you have very unique problems to solve, but is there still uh a potential or is there overly reliance?
• 23:27
  SPEAKER_01: Are people getting potentially getting lazy in terms of depending upon these tools or or offloading too much of these tools?
• 23:35
  SPEAKER_01: And and and if so, what's what's the implication of that?
• 23:38
  SPEAKER_02: Yeah, I I think there are from talking to our customers, I think there are probably two points that I would mention.
• 23:45
  SPEAKER_02: One is I don't think that there is an over-reliance just yet, right?
• 23:51
  SPEAKER_02: There may come a time where we do everything and nothing is done by humans anymore.
• 23:55
  SPEAKER_02: But so far, at least specifically for our unique quality of being able to tackle complex legacy code bases and large backlogs, the backlogs are huge, right?
• 24:06
  SPEAKER_02: But this is so people there is a lot of backlog tickets to churn through before people start running out of things to do.
• 24:11
  SPEAKER_02: And these were unnatural.
• 24:13
  SPEAKER_02: The second point I would say is you know, we we found a unique perspective from actually um an HR executive.
• 24:21
  SPEAKER_02: So we were talking to a large enterprise about deploying our solution on premises, which we actually managed to engineer a solution for.
• 24:27
  SPEAKER_02: Uh and you know, we are cloud native, and while main public product is a is a cloud product, but for this customer, we built an on-prem solution.
• 24:35
  SPEAKER_02: The the HR director uh or the SPP of HR essentially asked, how do we promote people if they are everyone's skill set is at parity?
• 24:46
  SPEAKER_02: When a junior engineer is doing the same work and the output as a senior engineer, what does a career ladder look like?
• 24:53
  SPEAKER_02: So I wouldn't necessarily call it lazy, right?
• 24:56
  SPEAKER_02: So we we didn't call people lazy uh you know, from using better IDEs or you know, better computers.
• 25:03
  SPEAKER_02: It's just a shift in technology and becoming more efficient.
• 25:06
  SPEAKER_02: Is my I'm maybe I'm a tech optimist here.
• 25:08
  SPEAKER_02: But I do think there are some consequences in terms of how do you assess, right?
• 25:12
  SPEAKER_02: Laziness is one factor, competency is another.
• 25:14
  SPEAKER_02: How do you assess the promotion factor?
• 25:16
  SPEAKER_02: And and prompt engineering may be it, right?
• 25:18
  SPEAKER_02: So you get the effectiveness of writing prompts is where you assess someone, or maybe you just look at sheer output, right?
• 25:23
  SPEAKER_02: Does it work?
• 25:24
  SPEAKER_02: And and what is a binary outcome?
• 25:26
  SPEAKER_02: Does it work and how many PRs did they they merge?
• 25:29
  SPEAKER_02: How many PRs they shipped what were merged in the end into the code base?
• 25:33
  SPEAKER_01: Interesting.
• 25:34
  SPEAKER_01: So I I was uh listening to an article on uh uh one of uh a podcast uh someplace, and they were talking about how uh organizations are uh uh trying to test people uh you know for employment.
• 25:49
  SPEAKER_01: And obviously one of the ways of of doing that is with task-based assessments.
• 25:54
  SPEAKER_01: And then the question is do you let people use AI tools or do you not let them use AI tools when they in fact will be using them in their work, right?
• 26:02
  SPEAKER_01: So it was kind of an interesting uh dilemma, and there was some uh discussion about how people are addressing that and and getting around that.
• 26:08
  SPEAKER_01: But I think your your point around evaluation is very important.
• 26:12
  SPEAKER_01: What and what what are the results?
• 26:14
  SPEAKER_01: What are the outcomes?
• 26:15
  SPEAKER_01: So, you know, when you think about the ability to handle autonomous development of very complex software, uh, you know, there's there's some really great thinking here that you've shared.
• 26:26
  SPEAKER_01: But what what are the what are the biggest misconceptions that you're seeing in the marketplace?
• 26:31
  SPEAKER_01: What are people not understanding about AI assisted or AI generated software engineering?
• 26:37
  SPEAKER_01: What are the misconceptions?
• 26:38
  SPEAKER_01: What are people missing?
• 26:40
  SPEAKER_02: I think the first one is in general, and maybe this is I shouldn't be doing it because I am in the industry.
• 26:45
  SPEAKER_02: I think in general, coding agents are overhyped.
• 26:47
  SPEAKER_02: I think that there is an expectation and you know, of being able to do tasks end-to-end fully, any task, anywhere, any code base.
• 26:56
  SPEAKER_02: We're not quite there yet, right?
• 26:58
  SPEAKER_02: I I think that there is no player, whether you are you know well funded or you know, lots of great talent, whether you are cognition in Devon, whether you are, you know, pool side, whatever you may be.
• 27:09
  SPEAKER_02: I don't believe anyone is there.
• 27:11
  SPEAKER_02: And I and I think that what the problem is, is that even though everyone claims to be a generalist coding agent, you do actually drill down to it.
• 27:19
  SPEAKER_02: There is a niche that people are kind of dominating.
• 27:22
  SPEAKER_02: So I I you know I may be misrepresenting my competitors and you know, fellow startup people, but you know, Cursor and Windsor, you know, Replit, right?
• 27:33
  SPEAKER_02: We count Amjan Massad as an investor, um, you know, are great at brand new projects.
• 27:39
  SPEAKER_02: But we we do think we are much more competitive.
• 27:41
  SPEAKER_02: We are lighter, we are faster, we are more capable at legacy backlog tasks that no engineer ever wants to be assigned to.
• 27:48
  SPEAKER_02: Maybe with time it will converge and we will become generally better as the models, underlying models become better as well.
• 27:55
  SPEAKER_02: But I think for now, the the the misconception is it's not fully capable.
• 27:59
  SPEAKER_02: We we you have to drill down and be honest and say you are you're better at specific things.
• 28:03
  SPEAKER_02: We we can, right?
• 28:05
  SPEAKER_02: If you if you go on to uh you know uh our our a website now and sign up, you you get a free trial, right?
• 28:12
  SPEAKER_02: So everyone can sign up and in and to try it out and you don't mind.
• 28:14
  SPEAKER_02: If you people don't like it, you can cancel.
• 28:16
  SPEAKER_02: But you can ask it to do uh do you need to do a brand new project, right?
• 28:19
  SPEAKER_02: So make a Tetris clone, you know, make me uh you know a data visualization website.
• 28:25
  SPEAKER_02: You can do it.
• 28:26
  SPEAKER_02: I just don't think that we are the best tool for that.
• 28:30
  SPEAKER_02: And I think there are plenty of other tools that are better.
• 28:33
  SPEAKER_01: So um interesting you talked about vibe coding uh earlier.
• 28:38
  SPEAKER_01: And what is your definition of vibe coding and and what do you think that means in in the marketplace?
• 28:45
  SPEAKER_01: And I imagine there's everything from you know, it it reminds me of the days of Lotus Notes of all things, back in the when people were sending their administrative assistants to application development classes, which I was teaching, saying, anybody can do this, right?
• 29:01
  SPEAKER_01: Anyone can to build the notes application.
• 29:03
  SPEAKER_01: And then you saw what it meant when administrative assistants built, you know, notes formed and so on.
• 29:08
  SPEAKER_01: But tell me what your thoughts are about about vibe coding, and then you know, what does it really mean in in your world, and then what do you think it means in the bigger picture?
• 29:19
  SPEAKER_02: Yeah, I where did that name come from?
• 29:21
  SPEAKER_02: I think I think for me, you know, and maybe my pause here gives you everything you need in terms of I think it's a fuzzy definition, and and I definitely bend the definition depending on the conversation I'm having here.
• 29:33
  SPEAKER_02: But I think in general I think it it is it is it is a seamless way for someone to get to turn their idea into something that is presentable to share with someone else.
• 29:46
  SPEAKER_02: It doesn't always have to work, and it doesn't, it definitely isn't seamless.
• 29:50
  SPEAKER_02: And you know, you've seen lots of horror stories now on X, where you know there's security vulnerabilities, there is the fact that you know after a certain point it's not scalable anymore and they have to rebuild it with a human engineer.
• 30:00
  SPEAKER_02: I think those problems will solve itself for time, right?
• 30:02
  SPEAKER_02: Everyone's improving, their product and models are improving.
• 30:05
  SPEAKER_02: But I think what it is for me, what vibe coding is, is where previously you had to use words and language to describe your idea, you can now show people your idea, right?
• 30:15
  SPEAKER_02: And that that time between you having a thought of an idea to actually be able to show people an idea has now reduced almost to z to zero because of vibe coding.
• 30:23
  SPEAKER_02: And and I think to some extent I agree, it has democratized.
• 30:26
  SPEAKER_02: You know, in our team, you know, we have people who are non-technical people who haven't learned to you know code.
• 30:31
  SPEAKER_02: And part of the Friday demo sessions that we have, you know, we have uh, you know, if we have some pizzas and we have some beers for everyone, we asked the non-technical people to build us uh a demo, right?
• 30:40
  SPEAKER_02: Whatever feature, whatever new idea they want, they have to prompt it their way.
• 30:43
  SPEAKER_02: We we helped them obviously set them up with with something like Vercel so that we can deploy and actually see it.
• 30:48
  SPEAKER_02: But we want we want to see that side.
• 30:50
  SPEAKER_02: And actually, I would say a lot of the time, maybe more than half of the time, the best idea that came out from that session is a non-technical person because they have a different perspective.
• 30:59
  SPEAKER_02: Whereas previously they could not possibly code that up.
• 31:03
  SPEAKER_01: Right.
• 31:03
  SPEAKER_01: And what what kinds of ideas are people coming up with?
• 31:06
  SPEAKER_01: And just just out of curiosity, like what would a vibe coding output be for a non-technical person in your organization?
• 31:14
  SPEAKER_02: Yeah, so you know, we uh we had I can come up with two ideas, right?
• 31:17
  SPEAKER_02: So we essentially had an idea which was sort of like an abandoned checkout uh idea.
• 31:22
  SPEAKER_02: So when you see a new customer go through the funnel and at some point they fall out.
• 31:29
  SPEAKER_02: So instead of emailing them and doing traditional mode, you can see certain behavior within the product already that you instantly insert a, let's say, a coupon, or give them an offer, or prompt them with a certain uh pop-up, or maybe something to nudge them in the right direction, where it was based on data so that you've gathered so far through the funnel, or people who are similar cohorts, right?
• 31:49
  SPEAKER_02: And that was an idea from a marketer where where you couldn't, you know, we probably an engineer may not have thought about that because they're not focused on it.
• 31:57
  SPEAKER_02: And and the way that they did it was to say, well, we wanted to build essentially um, you know, uh essentially a small app that was running internally that captured these data points and read it from somewhere like post hoc, where it caught that behavior and put them into a cohort, you would instantly give them an offer that you started pricing for them.
• 32:15
  SPEAKER_02: And another idea that what that came up was essentially, you know, based on again, if we found you know, certain behaviors or demographics to say, well, they may be more inclined to use to work on a project that is on a certain code base or coding language, or you know, they're from a certain maybe they're from a certain region where we say, okay, they're probably learning the language for the first time.
• 32:36
  SPEAKER_02: So JavaScript, TypeScript, maybe a more modern language, we kind of shift that again towards the the copy might change as as they are going through the just really great ideas that we would we we could action and generally will make an impact to our business.
• 32:50
  SPEAKER_02: Right.
• 32:50
  SPEAKER_01: No, that's great.
• 32:51
  SPEAKER_01: And I and I I totally see your point of giving those tools and capabilities to people who normally would not have them, but they still have great ideas and they can explore those ideas uh in a meaningful way.
• 33:02
  SPEAKER_01: So before we wrap up, when you look back at your career and your work across different sectors, what advice would you give to developers and technology uh professionals who are trying to make an impact in 2025 and beyond?
• 33:16
  SPEAKER_01: What kind of advice would you give people?
• 33:18
  SPEAKER_01: Sometimes I say, if you went back to yourself, if you could go back to yourself after you graduated college, what would you say?
• 33:25
  SPEAKER_01: That's one way to think about it.
• 33:26
  SPEAKER_01: But generally speaking, you know, where what guidance?
• 33:28
  SPEAKER_01: Because things are changing so rapidly.
• 33:30
  SPEAKER_01: And it's also kind of asked the question of where where the future careers will be and and so on.
• 33:36
  SPEAKER_01: But what kind of advice would you give people?
• 33:38
  SPEAKER_02: Yeah, I I I think again, speaking very kindly, I I always hesitate to give advice because I I still don't feel like I know what I'm doing either.
• 33:47
  SPEAKER_01: You're figuring it out as we all are.
• 33:49
  SPEAKER_02: I'm figuring it out as I go, and I feel like if I repeated my life again, you know, if it was a true controlled experiment, I'm not sure I would get the same result again.
• 33:57
  SPEAKER_02: So I I always I always hesitate because I always think there is a little bit of romanticizing for people who achieved any success to retrospectively, you know, uh apply logic to say, well, I was successful because I did this.
• 34:07
  SPEAKER_02: But I think I would encourage, and this is not unique, my younger self to be not afraid to take risks.
• 34:16
  SPEAKER_02: And and and I suppose I would frame that as in you kind of only need the dice to land on six once.
• 34:26
  SPEAKER_02: You know, you just need to go and do it, right?
• 34:29
  SPEAKER_02: Whatever thing that you have an inkling for, whatever that you you feel like this could work out or could not work out, particularly if you are young and you just graduated, you know, you don't have kids, you don't have a mortgage, you don't have that, just go do it.
• 34:40
  SPEAKER_02: Because like, yes, you may fail and you might be embarrassed, but actually if you just strike it lucky, you know, I've been very fortunate to have uh encountered and worked with and exited with lots of successful entrepreneurs.
• 34:53
  SPEAKER_02: And I think if you got them to speak candidly, they would say they worked hard, they are smart, but actually they got lucky, and that luck came from the fact that they actually did something, right?
• 35:03
  SPEAKER_02: This this idea of the bias towards action of just go and do it.
• 35:08
  SPEAKER_01: Right.
• 35:08
  SPEAKER_01: That's a great, that's a great point.
• 35:10
  SPEAKER_01: And you know, if you don't roll the dice, you're not gonna get six sixes, right?
• 35:14
  SPEAKER_01: So you have to take those risks and you have to take those chances.
• 35:17
  SPEAKER_01: And and you're right, you know, early in your career you can do that.
• 35:20
  SPEAKER_01: Well, thank you so much, Yang, for joining us today and sharing your thoughts and your insights.
• 35:25
  SPEAKER_01: I really appreciate it.
• 35:26
  SPEAKER_01: Um, and thanks to our listeners for tuning in.
• 35:28
  SPEAKER_01: And uh, of course, stay with us for more conversations about AI and and how things are evolving.
• 35:34
  SPEAKER_01: Also, want to thank uh Carolyn uh Southwick for uh her help in putting the show together.
• 35:39
  SPEAKER_01: But uh again, Yang, thank you so much for your time today.
• 35:42
  SPEAKER_01: Thank you for having me as well, Seth.
• 35:44
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 35:48
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 35:55
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
• 36:00
  SPEAKER_00: To learn more, visit early.com.
• 36:02
  SPEAKER_00: That's earlier.com.
• 36:06
  SPEAKER_00: Thank you for tuning in.