Latest / DevQuestions with Tim Corey / 208. Separating AI Hype from AI Reality

Transcript

208. Separating AI Hype from AI Reality · DevQuestions with Tim Corey

• 0:00
  SPEAKER_00: Welcome to the Dev Questions Podcast with Tim Corey.
• 0:03
  SPEAKER_00: Join us each episode as we tackle the questions you are asking about a career in software development, understanding the industry, and new technology.
• 0:12
  SPEAKER_00: If you're just starting out or you want to grow stronger as a developer, this is the place to get your questions answered.
• 0:19
  SPEAKER_00: Now, here's your host, expert developer and online educator Tim Corey.
• 0:25
  SPEAKER_02: How do I tell when AI products are real and when they're fake?
• 0:30
  SPEAKER_02: There's still a lot of questions around AI.
• 0:33
  SPEAKER_02: Seemingly every week we get a new announcement about new capabilities of one AI or another.
• 0:38
  SPEAKER_02: And some of these can seem like life-altering changes, like when the Devon team showed off their AI that entirely replaced a software developer.
• 0:49
  SPEAKER_02: Now, of course, it turns out that they faked a lot of the demo.
• 0:52
  SPEAKER_02: So, how do we determine what's real and what's fake in the AI space?
• 0:58
  SPEAKER_02: That's what we're going to talk about in today's episode of DevQuestions.
• 1:02
  SPEAKER_02: So let's start off with an understanding of what current AI is.
• 1:08
  SPEAKER_02: Because all AIs right now kind of come to the same route.
• 1:13
  SPEAKER_02: So let's talk about how they work and what they can and can't do.
• 1:18
  SPEAKER_02: Now, what we call AI currently is a system that's basically providing the next generation of a search engine.
• 1:26
  SPEAKER_02: You may say, Tim, that's not right.
• 1:27
  SPEAKER_02: No, they do more than that.
• 1:28
  SPEAKER_02: No, no, no, it's pretty much a search engine.
• 1:31
  SPEAKER_02: So let's talk about what I mean.
• 1:34
  SPEAKER_02: So the way AI is built, it takes a lot of inputs, and by a lot I mean billions, trillions, and so on, of inputs, and it breaks them down into small parts.
• 1:45
  SPEAKER_02: It observes how items connect to each other based upon how many times it sees those items connecting to each other in its training data.
• 1:54
  SPEAKER_02: For instance, it might see a million pictures of a maple tree and learn that maple trees have their leaves that typically connect at the end of branches, which then attaches to the trunk.
• 2:06
  SPEAKER_02: So it may see all this and say, okay, I see enough to know that, you know, if I see a maple leaf, it probably connects to a branch, and that branch probably goes back to a trunk and a trunk to the ground and so on.
• 2:19
  SPEAKER_02: It observes what it sees and sees those connections, and then it sees that over and over and over again and says, okay, I see how this could be.
• 2:30
  SPEAKER_02: So then, based upon that information and these associations, when we ask an AI to give us information, it pulls pieces from various connections that it has seen before and makes a new mixture of information based upon the perceived relevant data.
• 2:50
  SPEAKER_02: So that's the trap.
• 2:52
  SPEAKER_02: Okay.
• 2:53
  SPEAKER_02: So it takes these different correlations and hopefully that they're they have a true causation connection, but it creates a new mixture based upon the perceived relevant data and it gives it to us.
• 3:08
  SPEAKER_02: And again, that's the trap.
• 3:11
  SPEAKER_02: We see us as a new mixture and associate intelligence with the action, at least in our minds.
• 3:17
  SPEAKER_02: We see that the act in terms of a human act.
• 3:22
  SPEAKER_02: So we start to assume the AI is gaining in intelligence as it adds capacities.
• 3:29
  SPEAKER_02: So when we see an AI come back with this, you know, this paragraph of information that seemingly we couldn't find on the internet because it's not from one source, but it's from many as put it together.
• 3:41
  SPEAKER_02: Well, it's learning, right?
• 3:43
  SPEAKER_02: Like that's how we learn.
• 3:44
  SPEAKER_02: So we read a book or we read a web page or we watch a video and we absorb that information and then kind of put that together with other information we know and we put out a new conclusion.
• 3:57
  SPEAKER_02: So that's what the AI is doing, right?
• 4:00
  SPEAKER_02: Not really.
• 4:02
  SPEAKER_02: So what's that's not what's happening.
• 4:05
  SPEAKER_02: The AI is only mashing together things that is seen before into new combinations based upon the similarities to previously seen combinations.
• 4:16
  SPEAKER_02: You may say, well, Tim, it's still creating something new.
• 4:20
  SPEAKER_02: No, maybe.
• 4:22
  SPEAKER_02: Okay, yes, you may see some new things that maybe didn't exist anywhere, but that doesn't mean they're first of all right, because a lot of times they aren't.
• 4:31
  SPEAKER_02: And also it's based upon what it's seen before and just kind of regurgitating that, but in taking those smaller chunks and putting them together.
• 4:40
  SPEAKER_02: So it's kind of like instead of you know building a whole house, it's taking Lego pieces and putting them together and hoping that the result is a similar type house to what it's seen before.
• 4:52
  SPEAKER_02: That's why to this day, with the latest Chat GPT-4.0, which just came out a couple days ago, if you ask it to generate a nine-sided shape, it gives you shapes with six sides or a different count.
• 5:09
  SPEAKER_02: I tried just a minute ago and I kept telling it, no, that has six sides, try again.
• 5:14
  SPEAKER_02: No, that has six sides, try again.
• 5:16
  SPEAKER_02: And it's not figuring it out.
• 5:19
  SPEAKER_02: And if you are a toddler, one of the things you learn is how to count.
• 5:24
  SPEAKER_02: One, two, three, four.
• 5:26
  SPEAKER_02: That's pretty early on in the human learning.
• 5:29
  SPEAKER_02: But yet AI doesn't know how to count.
• 5:33
  SPEAKER_02: It doesn't count.
• 5:34
  SPEAKER_02: What it does is it takes previous information and repackages it.
• 5:39
  SPEAKER_02: So it's not reasoning.
• 5:41
  SPEAKER_02: It isn't counting the sides and going, wait, you're right.
• 5:44
  SPEAKER_02: Six sides.
• 5:45
  SPEAKER_02: In fact, as a human, when you look at that, you go, wait, one, two, three, four, five, six.
• 5:51
  SPEAKER_02: I see six, I don't see nine.
• 5:53
  SPEAKER_02: So you immediately know something's wrong.
• 5:56
  SPEAKER_02: The AI has given me this confidently saying, this is your answer.
• 6:01
  SPEAKER_02: And you say no, that's six sides, it will confidently give you a new one that also has six sides.
• 6:08
  SPEAKER_02: Now, in your example, it might be eight sides or it might be twelve sides, but for my examples over and over, I got six sides when I asked for nine.
• 6:18
  SPEAKER_02: Now, when I asked for three sides, did it every time.
• 6:21
  SPEAKER_02: There's a triangle.
• 6:22
  SPEAKER_02: There you go.
• 6:23
  SPEAKER_02: It even calls it a triangle.
• 6:24
  SPEAKER_02: If I give it a four-sided shape, yep, no problem.
• 6:28
  SPEAKER_02: There's there's four sides.
• 6:30
  SPEAKER_02: But nine-sided shapes are unusual.
• 6:33
  SPEAKER_02: It's not something that the AI does well because it's unusual.
• 6:39
  SPEAKER_02: It's not intelligent, it's just putting together what it's already seen.
• 6:43
  SPEAKER_02: And it hasn't seen a lot of nine-sided shapes.
• 6:47
  SPEAKER_02: So, with that understanding of AI, we now know the basics of what it can and cannot do.
• 6:56
  SPEAKER_02: It can create really specific searches for us.
• 7:00
  SPEAKER_02: So I love AI.
• 7:02
  SPEAKER_02: Don't get me wrong.
• 7:03
  SPEAKER_02: When I am developing something, I say, hey, how do I do a, I've been working recently to create different the HTTP verb commands in C sharp for a project I'm doing online.
• 7:16
  SPEAKER_02: Well, just asking it, hey, build this.
• 7:19
  SPEAKER_02: And I go, ah, it's not quite right.
• 7:20
  SPEAKER_02: Can you tweak this?
• 7:21
  SPEAKER_02: And it does.
• 7:22
  SPEAKER_02: That's super helpful.
• 7:24
  SPEAKER_02: It makes me faster because instead of going to Google and typing that question in, searching through a whole bunch of web pages, maybe Stack Overflow, getting the wrong answer a few times, or getting a piece from this web page and a piece from that web page and kind of piecing together, it's doing the piecing together for me.
• 7:42
  SPEAKER_02: Instead of giving me whole web pages, it's giving me specific parts to my solution and piecing them together for a result that's tailored for me.
• 7:52
  SPEAKER_02: Now, the downside is that that piecing together might not be correct.
• 7:58
  SPEAKER_02: And if there isn't enough training examples to allow it to generate a similar answer to an existing solution, it's more likely to be incorrect.
• 8:06
  SPEAKER_02: It's going to assume things.
• 8:08
  SPEAKER_02: And as we know as software developers, assumptions are a really bad thing.
• 8:14
  SPEAKER_02: Now that's what AI can do.
• 8:16
  SPEAKER_02: It can be a really specific search engine and piece things together in a much better way than just a Google search can.
• 8:23
  SPEAKER_02: But what AI cannot do is generate new answers on its own.
• 8:29
  SPEAKER_02: It's not inventing, it's not creating something new.
• 8:33
  SPEAKER_02: You may say, well, Tim, it's piecing together new answers.
• 8:36
  SPEAKER_02: Again, yes and no.
• 8:38
  SPEAKER_02: What it's doing is taking existing answers, small chunks of them, and putting them together.
• 8:43
  SPEAKER_02: It's not building new answers.
• 8:45
  SPEAKER_02: It's just taking existing answers in smaller parts and putting them together and giving them to you.
• 8:52
  SPEAKER_02: So it can't invent or create.
• 8:55
  SPEAKER_02: The only seemingly new things come from when it has seen something similar beforehand, unless you count the remix, which is its best guess at a solution.
• 9:07
  SPEAKER_02: So at the very most, what it's doing is taking small chunks of things from other people that people have created and then putting together.
• 9:17
  SPEAKER_02: That's the best that you can call creation, but it's not really.
• 9:22
  SPEAKER_02: So now let's talk about what how to differentiate hype from reality when it comes to artificial intelligence.
• 9:29
  SPEAKER_02: We can call anything hype that involves an AI creating something completely new.
• 9:36
  SPEAKER_02: We can also identify something as hype if it involves autonomous actions by an AI.
• 9:44
  SPEAKER_02: Autonomy requires logic.
• 9:47
  SPEAKER_02: AIs don't possess logic, nor can they.
• 10:04
  SPEAKER_02: So when Cognition introduced Devon, the in their term words, the world's first AI developer, your hype radar should have been going off like crazy.
• 10:17
  SPEAKER_02: Being a software developer isn't primarily about writing code.
• 10:21
  SPEAKER_02: We talked about that, I think, last week.
• 10:23
  SPEAKER_02: It's about building logic, understanding a specific situation, and building something that fits for that situation.
• 10:31
  SPEAKER_02: That's why senior software developers so often use the phrase, it depends, when you ask them for advice.
• 10:39
  SPEAKER_02: Just throwing code at a problem isn't the solution.
• 10:43
  SPEAKER_02: So what would Devin have to do as a software developer?
• 10:49
  SPEAKER_02: Well, it'd have to design a solution specific to the situation.
• 10:54
  SPEAKER_02: For example, the right language, work in an existing code base, have current data, and use that current data because it might, that current data might change how you design something.
• 11:05
  SPEAKER_02: Figure out how to work inside a specific industry.
• 11:07
  SPEAKER_02: Maybe that's how you do certain security things or something else.
• 11:12
  SPEAKER_02: Number two, it would have to understand the needs of the customer, including knowing when to ask follow-up questions.
• 11:20
  SPEAKER_02: That's really important.
• 11:22
  SPEAKER_02: Knowing how to ask a follow-up question because maybe the user thinks they want something, but they don't realize what they're asking for.
• 11:30
  SPEAKER_02: Devin can't think that through logically and go, you know what?
• 11:34
  SPEAKER_02: You're probably asking for this, but what you really want is that.
• 11:39
  SPEAKER_02: Okay, knowing that how to ask that follow-up question, that involves logic processing and something that AI cannot do.
• 11:49
  SPEAKER_02: And also, number three, it know it has to know how to evaluate if the code does what it was asked to do.
• 11:58
  SPEAKER_02: Okay, so when you get requirements, you have to be able to say, did I meet the requirements?
• 12:04
  SPEAKER_02: Not just do the unit tests pass.
• 12:07
  SPEAKER_02: That doesn't really matter in the grand scheme of things.
• 12:10
  SPEAKER_02: If you ask me to create a method that added two numbers together, and I create a method that multiplied three numbers together, and I create a unit test to make sure that yes, it does multiply the three numbers together, the unit test is going to pass, but I didn't do what you asked me to do.
• 12:28
  SPEAKER_02: So I failed the assignment, even though I have working code.
• 12:33
  SPEAKER_02: So knowing how to evaluate your requirements and then evaluate your solution to make sure it matches the requirements is something that AI just can't do at this point.
• 12:44
  SPEAKER_02: All these tasks involve logic and intelligence.
• 12:49
  SPEAKER_02: Remember, that's not what AI does.
• 12:52
  SPEAKER_02: And yet when Devon was launched, they showed it solving problems in code.
• 12:58
  SPEAKER_02: They showed it learning from documentation.
• 13:01
  SPEAKER_02: They even showed it performing tasks on Upwork as if it were a real software developer.
• 13:07
  SPEAKER_02: And that's when you should know, should have known that, yeah, this is hype and this is not real.
• 13:12
  SPEAKER_02: Okay.
• 13:13
  SPEAKER_02: Now, if you look closely at the demos, you're gonna see a few things.
• 13:16
  SPEAKER_02: The Upwork task that Devin says it completed, it didn't even result in a solution that met the initial requirements.
• 13:24
  SPEAKER_02: Again, coming back to can it evaluate its existing code against what it was asked to do?
• 13:30
  SPEAKER_02: No.
• 13:31
  SPEAKER_02: Now, of all the things that AI can't do yet, I think that's the easiest one to maybe come closer to, but even that isn't possible right now with AI.
• 13:41
  SPEAKER_02: Uh, number two, the problem that Devin identified.
• 13:45
  SPEAKER_02: So there's a video where he the AI identifies a problem, it debugs it, puts some print line statements in, and then it fixes it.
• 13:52
  SPEAKER_02: Well, the problem is those problems were actually caused by Devin.
• 13:59
  SPEAKER_02: It caused the errors because it couldn't figure out how to clone the repository and build the code, even though there was clear instructions that were included in the repository.
• 14:09
  SPEAKER_02: Instead, it created extra files that it then had to debug in order to try to get those to a working state when the files weren't needed in the first place.
• 14:20
  SPEAKER_02: So it caused the problems that it then is proud of fixing.
• 14:25
  SPEAKER_02: I mean, that's what we do as software developers too sometimes, but that doesn't really show any type of value as a tool.
• 14:34
  SPEAKER_02: Because it didn't think anything through logically, it started throwing code at solution when that wasn't the way to fix it.
• 14:42
  SPEAKER_02: And then number three, in their Lama 2 benchmark demo, they said that they were turning over the control to Devon and that Devin's gonna be in the driver's seat.
• 14:51
  SPEAKER_02: However, you can repeatedly see an operator correcting Devin and redirecting it throughout the process.
• 14:59
  SPEAKER_02: Basically, the operator is providing the reason and logic, and Devin is just attempting to write code, which means that the operator was doing a software development role and Devin was doing kind of a co-pilot role.
• 15:15
  SPEAKER_02: So it turns out that the Devin demo was faked and it didn't actually perform much differently than GitHub co-pilot or chat GPT.
• 15:25
  SPEAKER_02: So that's something that you have to think through is what can the AI do and what can't it do?
• 15:31
  SPEAKER_02: And is some of this going to be fake or hype as opposed to reality?
• 15:37
  SPEAKER_02: Everyone is trying to get in a gold rush of AI.
• 15:41
  SPEAKER_02: There's going to be a lot of misinformation out there.
• 15:44
  SPEAKER_02: So you have to be able to evaluate what could it possibly do.
• 15:48
  SPEAKER_02: There is a lot that AI can do, but being able to evaluate the hype versus the reality is really important.
• 15:57
  SPEAKER_02: Now, now that you have an understanding of what AI is, how it works, and what it can't do, hopefully you'll be better equipped to separate the hype from the reality of AI.
• 16:08
  SPEAKER_02: And there are a lot of great things coming out with AI.
• 16:12
  SPEAKER_02: So, for example, near real-time language translation, that's awesome.
• 16:17
  SPEAKER_02: Better code writing assistance, it's amazing.
• 16:20
  SPEAKER_02: I use it a lot.
• 16:22
  SPEAKER_02: And incredible web search capabilities can really make a huge difference in our lives.
• 16:27
  SPEAKER_02: AI can make our lives better.
• 16:31
  SPEAKER_02: It just won't replace the true purpose of a software developer.
• 16:36
  SPEAKER_02: Now, it will, however, make software developers more efficient and effective.
• 16:42
  SPEAKER_02: So that's the other part of the fear of AI is well, it's going to replace software developers.
• 16:48
  SPEAKER_02: No, absolutely not.
• 16:49
  SPEAKER_02: It can't do logic.
• 16:50
  SPEAKER_02: Okay.
• 16:51
  SPEAKER_02: So, well, but software developers are going to use AI to be more efficient, therefore to be less software developers.
• 16:58
  SPEAKER_02: Kind of, yes.
• 17:00
  SPEAKER_02: Because software developers can do more things because they have an AI assistant.
• 17:06
  SPEAKER_02: That's always been true of our tooling, though.
• 17:09
  SPEAKER_02: When basic, not visual basic, when basic first came out, people thought that the career of software developer, which was barely a thing, they thought it was dead because anybody in their basement could write code.
• 17:24
  SPEAKER_02: And so they thought that by making basic, that it would essentially eliminate the field of software development.
• 17:33
  SPEAKER_02: Going back even further, when they made assembly, they thought the same thing.
• 17:39
  SPEAKER_02: It's going to eliminate software development because it's so easy now to write applications.
• 17:45
  SPEAKER_02: These things didn't happen.
• 17:47
  SPEAKER_02: In fact, software development has been on a hockey stick curve of growth.
• 17:52
  SPEAKER_02: It's been booming.
• 17:54
  SPEAKER_02: So, yes, there will be some more efficiencies added to developers that AI will help with because AI is valuable, but that doesn't mean it's going to destroy the market for developers or even new developers.
• 18:10
  SPEAKER_02: Because it actually makes new developers, you know, it allows them to get in the market more easily.
• 18:17
  SPEAKER_02: So what it will do is it will open up new jobs and new possibilities and new, allow smaller companies to bring on software developers and really grow how many different areas can employ software developers.
• 18:33
  SPEAKER_02: So it really is actually a good thing for the industry.
• 18:36
  SPEAKER_02: So instead of being centralized in a few companies, which I know we're not, but everyone thinks that if you don't work at Google or Facebook or actually Meta or Microsoft or one of those, that you're not actually a software developer.
• 18:50
  SPEAKER_02: I don't know.
• 18:51
  SPEAKER_02: But people look at those companies and say, well, they're laying off, therefore the industry is dying.
• 18:55
  SPEAKER_02: That's not how that works.
• 18:56
  SPEAKER_02: There's so many different companies that hire software developers.
• 18:59
  SPEAKER_02: With AI, there'll be even more, which means it diversifies the industry.
• 19:04
  SPEAKER_02: It allows for more possibilities of working in more places.
• 19:10
  SPEAKER_02: So AI is a good thing.
• 19:12
  SPEAKER_02: AI is going to be helpful, but we have to make sure we don't get on the hype train and believe that AI is somehow more than it is.
• 19:21
  SPEAKER_02: So that's how we separate the hype from the reality when it comes to artificial intelligence.
• 19:27
  SPEAKER_02: Thanks for the question.
• 19:28
  SPEAKER_02: And as always, I am Tim Corey.
• 19:31
  SPEAKER_01: Thank you for joining us for this episode of Deaf Questions.
• 19:34
  SPEAKER_01: Tim is committed to making it easier for you to become a developer.
• 19:37
  SPEAKER_01: If you would like to help make more content like this possible, please like, subscribe, rate, and share Deaf Questions.
• 19:45
  SPEAKER_01: You can also send your questions to questions at IamTimcore.com.
• 19:50
  SPEAKER_01: Until next time, remember, you are too smart and your time too valuable to waste it making all the mistakes Tim did.
• 19:57
  SPEAKER_01: When you're ready to learn to think and code like a professional developer, head over to imtimcore.com and enroll in a course.