Latest / DevQuestions with Tim Corey / 319. How To Properly Evaluate a Technology

Transcript

319. How To Properly Evaluate a Technology · DevQuestions with Tim Corey

• 0:00
  SPEAKER_01: The most used phrase of a senior software developer should be it depends.
• 0:05
  SPEAKER_01: That's because every situation is different, and a solution to any problem needs to take a number of situation-specific variables into account.
• 0:14
  SPEAKER_01: That's why there are so many languages, frameworks, and ways of doing things in software development.
• 0:20
  SPEAKER_01: But what we don't talk about as much is how to properly evaluate an option or a technology.
• 0:26
  SPEAKER_01: In today's episode of DevQuestions, we're going to discuss how to properly evaluate your options without letting enthusiasm, pessimism, ignorance, or inertia get in the way.
• 0:49
  SPEAKER_01: Software development is more than just writing code.
• 0:52
  SPEAKER_01: So let's talk about the rest of it.
• 0:54
  SPEAKER_01: Specifically, let's talk about properly evaluating your options.
• 0:59
  SPEAKER_01: And let's start by looking at the four major categories of people.
• 1:03
  SPEAKER_01: Number one is the enthusiast.
• 1:05
  SPEAKER_01: So this is a person who knows all the new features of a thing, how it's better, how it's improvement, what's changed.
• 1:15
  SPEAKER_01: They are very enthusiastic about this new thing.
• 1:20
  SPEAKER_01: Now, you can kind of overlay that on an ad, right?
• 1:25
  SPEAKER_01: So if if you heard the maybe the head of an LLM talk about their new thing, it's going to sound an awful lot like an enthusiast.
• 1:34
  SPEAKER_01: That's not to say that enthusiasts are just people spouting ads.
• 1:38
  SPEAKER_01: They're not.
• 1:39
  SPEAKER_01: But what they've done is they have looked at all these new things and they're excited for it.
• 1:45
  SPEAKER_01: That's the enthusiast.
• 1:47
  SPEAKER_01: Number two is the pessimist.
• 1:49
  SPEAKER_01: So this is the person who's going to spot all the problems.
• 1:54
  SPEAKER_01: And they might sound a little depressed, right?
• 1:56
  SPEAKER_01: So they they talk about, oh, this new technology, yeah, that that thing's got a whole bunch of security holes, probably.
• 2:03
  SPEAKER_01: It's probably not written real well.
• 2:04
  SPEAKER_01: You know what?
• 2:05
  SPEAKER_01: It's gonna be the end of civilizations.
• 2:06
  SPEAKER_01: We know it.
• 2:07
  SPEAKER_01: Here's all the downsides why it's going to fail.
• 2:10
  SPEAKER_01: Now, again, this person isn't necessarily wrong.
• 2:15
  SPEAKER_01: They're going to spot the issues, the problems, the downsides with this thing, whatever this new thing is.
• 2:24
  SPEAKER_01: Number three is the ignorer.
• 2:27
  SPEAKER_01: This is the person who hasn't really looked into it.
• 2:30
  SPEAKER_01: You know what, not really passionate either way.
• 2:33
  SPEAKER_01: Give it time.
• 2:34
  SPEAKER_01: You know, it's the person that, you know, yeah, I heard it's out there, but that's about it.
• 2:39
  SPEAKER_01: I haven't looked into it.
• 2:40
  SPEAKER_01: I kind of just left it off the radar.
• 2:43
  SPEAKER_01: I'm just kind of ignoring it.
• 2:45
  SPEAKER_01: And this person is in their own way right as well, because they're not just chasing down the latest trend.
• 2:54
  SPEAKER_01: They're not always disrupting everything.
• 2:56
  SPEAKER_01: They're kind of picking and choosing when and kind of waiting for more of the tried and true over the bleeding edge.
• 3:05
  SPEAKER_01: And number four type of personality is the unmoving person.
• 3:10
  SPEAKER_01: This is the person who's stuck in inertia.
• 3:12
  SPEAKER_01: They're stuck in the rut, where the way they've done things is the way they're going to continue to do things because it works.
• 3:19
  SPEAKER_01: If it doesn't, you know, if it's not broken, then don't fix it, right?
• 3:24
  SPEAKER_01: That's the type of person that this person is.
• 3:26
  SPEAKER_01: Um, the old ways are the best ways.
• 3:28
  SPEAKER_01: You know, these new things, they're they're wrong for these reasons.
• 3:32
  SPEAKER_01: The old way was the best way of doing things.
• 3:33
  SPEAKER_01: You know what?
• 3:34
  SPEAKER_01: We should still be programming in whatever language.
• 3:37
  SPEAKER_01: I don't want to offend anybody, but you know, the the older languages or the older frameworks or this way was best.
• 3:43
  SPEAKER_01: And, you know, they're not wrong either because change is costly.
• 3:48
  SPEAKER_01: And so there is something to be said for not just changing all the time.
• 3:53
  SPEAKER_01: So here's the truth: all four people or personality groups are correct.
• 4:00
  SPEAKER_01: They just aren't fully correct.
• 4:03
  SPEAKER_01: You need all four to come together to evaluate a new option.
• 4:08
  SPEAKER_01: Too often, people are afraid to face the other side of an argument, where they either like it or hate it, and they want to find things that affirm that rather than confronting the fact that there is another side.
• 4:24
  SPEAKER_01: And that fear can really hold you back.
• 4:27
  SPEAKER_01: If you cannot confidently and honestly face the other side of an argument, then you don't have a strong argument.
• 4:35
  SPEAKER_01: What you have is just the thing that you like.
• 4:38
  SPEAKER_01: And you can like things, but it doesn't mean that you're making an honest argument.
• 4:43
  SPEAKER_01: So let's be clear: if you cannot properly and fully fill out all four roles for any decision you make, you aren't making good decisions.
• 4:54
  SPEAKER_01: You're just hoping for a happy accident.
• 4:58
  SPEAKER_01: People often say they're evaluating the pros and cons, but they're often lying to themselves.
• 5:03
  SPEAKER_01: The most common situation is the person is fully in one category and kind of pretending you're making a half effort in the others.
• 5:11
  SPEAKER_01: Let's go through a relevant example because this has come up a lot recently, and that is AI.
• 5:17
  SPEAKER_01: Everyone has an opinion on AI.
• 5:20
  SPEAKER_01: I've had conversations with each of the four categories of people about AI.
• 5:24
  SPEAKER_01: So let's start with each category and what it might look like when it comes to AI, and then we'll narrow it down a bit to be just AI for developers because AI is such a broad category.
• 5:34
  SPEAKER_01: So let's talk about the enthusiast.
• 5:36
  SPEAKER_01: You know, AI makes me write code faster.
• 5:39
  SPEAKER_01: I get work done, I get more work done, I get to the market quicker, I do more with less.
• 5:44
  SPEAKER_01: This is the enthusiast.
• 5:45
  SPEAKER_01: They they see all the benefits of being able to have the code written for you where you just say, I want this, and it pops out the other end.
• 5:55
  SPEAKER_01: It can get a lot of work done fast.
• 5:58
  SPEAKER_01: We see a lot of this.
• 6:00
  SPEAKER_01: And you know what?
• 6:01
  SPEAKER_01: There is something said for the fact that, yes, it can make you a faster developer.
• 6:07
  SPEAKER_01: It can help you write code better, you can get more work done with AI.
• 6:12
  SPEAKER_01: And by the way, pause right here.
• 6:14
  SPEAKER_01: If you are against AI or frustrated AI, which I can I can get, but if you can't say honestly that there's positives to it, then again, you're you're just looking at one side.
• 6:26
  SPEAKER_01: You've got to look at both sides.
• 6:28
  SPEAKER_01: Now, that doesn't mean that you think that these are all gonna be the reason to do it.
• 6:34
  SPEAKER_01: You just need to acknowledge the fact that there are reasons why it exists.
• 6:39
  SPEAKER_01: There's a reason why it's positive.
• 6:40
  SPEAKER_01: Okay, so that's the enthusiast.
• 6:42
  SPEAKER_01: It writes code faster, it gets more work done, you can get to market quicker, you can do more with less.
• 6:48
  SPEAKER_01: It can empower companies in a way that smaller businesses can actually get into the market and do custom things as opposed to just buying off the shelf and being limited.
• 6:59
  SPEAKER_01: So there's a lot of reasons you can list, and I've only listed a few, but why you can be enthusiastic about AI?
• 7:07
  SPEAKER_01: Let's look at the pessimist.
• 7:09
  SPEAKER_01: Well, AI coding can introduce more bugs because you're moving faster, and so you miss things.
• 7:15
  SPEAKER_01: And AI doesn't always write the correct code.
• 7:19
  SPEAKER_01: It writes it correct a lot of the time, but not all the time.
• 7:22
  SPEAKER_01: And so it can introduce some significant bugs.
• 7:26
  SPEAKER_01: Also, you lose institutional knowledge about the code, meaning when the AI writes it and you're just reviewing it, well, then you're not really baked into why the decisions were made.
• 7:36
  SPEAKER_01: So you're no longer the go-to person for the code.
• 7:40
  SPEAKER_01: You might say, well, AI is.
• 7:41
  SPEAKER_01: Well, yeah, but AI doesn't remember, right?
• 7:44
  SPEAKER_01: Where humans, we've gotten used to this idea that there's a senior developer who knows where all the bodies are buried, right?
• 7:50
  SPEAKER_01: They they know why we made these decisions and why we made these compromises.
• 7:55
  SPEAKER_01: And you don't know that as much.
• 7:57
  SPEAKER_01: So you also don't have as good an understanding about your security and performance.
• 8:01
  SPEAKER_01: And AI is not great at that usually.
• 8:04
  SPEAKER_01: So that's gonna be a weak point for your company.
• 8:08
  SPEAKER_01: Again, if you are an AI enthusiast and going, yep, yep, yep, well, here's the thing.
• 8:16
  SPEAKER_01: You need an, I'm sorry, AI, a negative person towards AI.
• 8:19
  SPEAKER_01: And you're going, yes, yes, yes.
• 8:20
  SPEAKER_01: Well, but remember, there's a positive as well.
• 8:23
  SPEAKER_01: Okay, we have to understand both sides.
• 8:26
  SPEAKER_01: So if you're an enthusiast and go, you know, those are those are not problems, well, then you're missing out.
• 8:32
  SPEAKER_01: There are problems.
• 8:33
  SPEAKER_01: You have to acknowledge it.
• 8:34
  SPEAKER_01: Um, this is one of the reasons why I did the AI hurts everything series.
• 8:38
  SPEAKER_01: It wasn't just to be a pessimist, which is more of what those videos are about, is the pessimistic side, not because I'm a pessimist, but because I was kind of fearful that people weren't looking at that side of things, where they were looking at the enthusiast side, but not the pessimist side.
• 8:56
  SPEAKER_01: And yes, you have to look at both of those.
• 8:58
  SPEAKER_01: But again, there's two more to go through.
• 9:01
  SPEAKER_01: Number three, the ignorer.
• 9:03
  SPEAKER_01: So, you know, if if you're looking at AI and saying, hey, I want to look at from the role of the ignorer, well, I'm gonna look at this and go, you know what?
• 9:11
  SPEAKER_01: We need to wait for some dust to settle right now.
• 9:14
  SPEAKER_01: Because the way to use AI today is not the way we used AI a month ago, which is not the way we used AI two months ago.
• 9:27
  SPEAKER_01: I mean, are you doing loops right now?
• 9:29
  SPEAKER_01: Are you doing uh agents?
• 9:31
  SPEAKER_01: Are you doing skills?
• 9:33
  SPEAKER_01: Are you doing MCP?
• 9:34
  SPEAKER_01: Are you doing like all these things are changing and constantly evolving?
• 9:39
  SPEAKER_01: And the problem is the way you do things today won't work the same in a month.
• 9:45
  SPEAKER_01: The prompt you write today that's that's perfectly customized for that version of the LM won't work as well next month.
• 9:53
  SPEAKER_01: So, yeah, there's a little bit of thinking about, you know what, maybe you should wait for the dust, thus, the dust to settle a bit on some of this.
• 10:00
  SPEAKER_01: Maybe we should think about what we do and don't do when it comes to AI because of how fluid it is.
• 10:08
  SPEAKER_01: Maybe we skip some of the churn of that new technology and we wait for the evidence to mount over what is the right way of doing things.
• 10:17
  SPEAKER_01: Maybe we let other people kind of get ahead and kind of learn the ropes, figure it all out, and then we come along and build on their shoulders instead of trying to replicate them.
• 10:30
  SPEAKER_01: So the ignorer has some good points as well, but you know what?
• 10:34
  SPEAKER_01: There's a lot of churn going on, and it's being expensive for us to jump in right now.
• 10:39
  SPEAKER_01: It's even more expensive because of the cost of the credits.
• 10:42
  SPEAKER_01: So if we ignore and don't just try and learn as we go and reinvent the wheel like everybody else is doing, we can just learn from others' experiences and not have to pay for them.
• 10:56
  SPEAKER_01: And number four, the unmoving.
• 10:59
  SPEAKER_01: Still doing things the way we used to, you know, that that could be something that is beneficial.
• 11:06
  SPEAKER_01: Because maybe if we continue to do things the way we used to, if the whole market changes, but it doesn't get better.
• 11:12
  SPEAKER_01: So we're seeing things like there's more security flaws and there's worse performance, and you know, companies are struggling, and companies are out of business because of the push to go fast.
• 11:24
  SPEAKER_01: Well, if you're unmoving, you're still doing the same way you used to do things, you might actually move ahead of the pack because now you're the reliable one.
• 11:35
  SPEAKER_01: And this allows you to say, you know what?
• 11:38
  SPEAKER_01: Maybe we do things the way we've done things because we've proven that it works.
• 11:43
  SPEAKER_01: And it's if something that, you know, it wasn't broken.
• 11:47
  SPEAKER_01: So let's not try and meddle with it.
• 11:50
  SPEAKER_01: Let's keep our same processes and technologies.
• 11:53
  SPEAKER_01: Now, again, remember, we're not looking at just one of those people.
• 11:58
  SPEAKER_01: We want to look at all four then.
• 12:02
  SPEAKER_01: So the reality is that too many people could only passionately argue for one of those categories.
• 12:08
  SPEAKER_01: The rest they only give passing lip service to.
• 12:11
  SPEAKER_01: So if you're making a decision about AI for development, you should be able to answer the following questions.
• 12:17
  SPEAKER_01: Number one, what benefits will this actually add to our team and process?
• 12:22
  SPEAKER_01: I want you to stop right there.
• 12:24
  SPEAKER_01: We're not talking about what are the talking points.
• 12:26
  SPEAKER_01: We're not talking about what are the cool things about AI.
• 12:30
  SPEAKER_01: Nope.
• 12:30
  SPEAKER_01: We're talking about what benefits will this actually add to our team or process?
• 12:37
  SPEAKER_01: What's the real concrete things that's going to add?
• 12:39
  SPEAKER_01: And you're going to find some.
• 12:41
  SPEAKER_01: You probably should.
• 12:42
  SPEAKER_01: If you don't, stop right there because you're done, right?
• 12:45
  SPEAKER_01: But number two, what downsides is this going to add to our team or process?
• 12:51
  SPEAKER_01: And again, you should know specifically for your environment.
• 12:56
  SPEAKER_01: Uh, really quickly, one you can add is there's a cost, right?
• 13:00
  SPEAKER_01: Nothing's free.
• 13:01
  SPEAKER_01: And AI is not cheap and it's not getting cheaper.
• 13:04
  SPEAKER_01: It's getting more expensive.
• 13:06
  SPEAKER_01: So that's one downside, but that's not the only downside.
• 13:10
  SPEAKER_01: But if you have to look at how is it being used in the industry and what are the results we're seeing, and go, oh, there's some other downsides.
• 13:17
  SPEAKER_01: That's why I did the video on how it hurts software development, because there are some impacts that people aren't addressing.
• 13:25
  SPEAKER_01: So, what are the downsides to this?
• 13:26
  SPEAKER_01: We have the benefits listed.
• 13:28
  SPEAKER_01: Now do we have the downsides to kind of put on that scale to say this is starting to weigh more or less than the other?
• 13:36
  SPEAKER_01: Now we're not done yet, though, because number three, has this been tested or are we rushing in?
• 13:44
  SPEAKER_01: So have we are we implementing a tested thing where people have said this is how you do things?
• 13:51
  SPEAKER_01: Or is it this is how you do things this week, and we're gonna have to change a process over and over again?
• 13:57
  SPEAKER_01: And do we even have the bandwidth to change right now?
• 14:01
  SPEAKER_01: If you're in the middle of a process, you're middle of a system, upgrading just isn't the right time.
• 14:07
  SPEAKER_01: You know, doing things differently halfway through a major project isn't the right time.
• 14:12
  SPEAKER_01: A very simple example of this is I use recording software.
• 14:16
  SPEAKER_01: I'm using recording software right now to record myself.
• 14:19
  SPEAKER_01: Well, I get updates all the time.
• 14:21
  SPEAKER_01: Unfortunately, it's faster and faster now, where it will say, hey, there's a new update.
• 14:26
  SPEAKER_01: There's a new update.
• 14:27
  SPEAKER_01: Guess what I don't do?
• 14:29
  SPEAKER_01: I don't upgrade right away.
• 14:31
  SPEAKER_01: I don't make changes until I know I have a day or two to figure out the problems.
• 14:37
  SPEAKER_01: Because if I say every time I see an upgrade, yeah, go ahead and do that, and then jump right into recording.
• 14:43
  SPEAKER_01: I've had times where I've done that and then I have no sound because I change the default uh capture, or the video looks weird, or I have a sync issue.
• 14:54
  SPEAKER_01: So you want to make sure that you are using something that's been tested and that you have the bandwidth to do the change right now.
• 15:03
  SPEAKER_01: That's number three.
• 15:04
  SPEAKER_01: Number four, does this new system improve our process significantly enough?
• 15:12
  SPEAKER_01: Not does it make an improvement, is it significant enough?
• 15:17
  SPEAKER_01: It has to overcome that inertia.
• 15:20
  SPEAKER_01: You have a process.
• 15:22
  SPEAKER_01: In theory, it's working.
• 15:24
  SPEAKER_01: In theory, you're putting software out there.
• 15:26
  SPEAKER_01: So is this change going to make your process significantly better?
• 15:34
  SPEAKER_01: Okay.
• 15:34
  SPEAKER_01: Is this going to provide enough gain for the pain that's going to cause?
• 15:39
  SPEAKER_01: Because any new change is going to cause pain.
• 15:42
  SPEAKER_01: So you should know the answers to all these questions before you make an informed decision.
• 15:48
  SPEAKER_01: Otherwise, it's not an informed decision, it's a wishful guess.
• 15:54
  SPEAKER_01: So once you've honestly evaluated all four possibilities, can you, or all four positions, I guess you'd say, um, you can make an informed decision about what to do, but you aren't done.
• 16:06
  SPEAKER_01: Then you need to evaluate if you thought what was going to be true actually came true, right?
• 16:13
  SPEAKER_01: So you can't stop it, we made a decision.
• 16:15
  SPEAKER_01: You have to wait a month, two months, three months down the road, and then look back at those four questions that you answered and say, were we right about these answers?
• 16:27
  SPEAKER_01: Did the good things come true?
• 16:29
  SPEAKER_01: Did the bad things happen at the level we expected them to?
• 16:33
  SPEAKER_01: Did this benefit us much more than what we were currently going through?
• 16:38
  SPEAKER_01: You know, answering these questions later and seeing, were we correct in our assumptions?
• 16:44
  SPEAKER_01: Because there are assumptions.
• 16:46
  SPEAKER_01: So are you only experiencing the downsides you expected?
• 16:50
  SPEAKER_01: You know, are you not getting the benefits?
• 16:52
  SPEAKER_01: Like you need to understand how to evaluate and see, you know, just because you're doing it now doesn't mean you should keep doing it.
• 17:00
  SPEAKER_01: Maybe you look at it and go, oops, we were wrong.
• 17:04
  SPEAKER_01: Okay?
• 17:05
  SPEAKER_01: So this is important to understand.
• 17:07
  SPEAKER_01: We often allow our bias to cloud our judgment, but that way leads to poor decisions and guesses.
• 17:14
  SPEAKER_01: Instead, you need to actively push against your bias to get to a point where you can make well-informed, honest evaluations that can also adjust over time.
• 17:26
  SPEAKER_01: This is one of the where areas where I was very successful as a consultant because of the fact that I could say, hey, we were wrong.
• 17:35
  SPEAKER_01: Hey, you know what?
• 17:36
  SPEAKER_01: We we put that in, it didn't work the way you expect it to.
• 17:39
  SPEAKER_01: Let's make an adjustment.
• 17:40
  SPEAKER_01: Or here's what I've learned over time, the positives and the negatives.
• 17:45
  SPEAKER_01: And that really can have a positive impact on a department because you're coming in and showing the full spectrum.
• 17:54
  SPEAKER_01: Because you know what, if you're super enthusiastic about something and you say, you know what, this new widget is going to do amazing things.
• 18:01
  SPEAKER_01: Here's all the benefits it can do.
• 18:03
  SPEAKER_01: And you forget to evaluate the rest of the stuff and you implement it, the rest of stuff's gonna show up.
• 18:10
  SPEAKER_01: It's not going to be something where you put it in all of a sudden everything's sunshine and roses because you said all these new positive things.
• 18:18
  SPEAKER_01: The negative things come along, whether you want them to or not.
• 18:23
  SPEAKER_01: So if you're informed and you inform people about that, they at least know up front, this is what we're expecting for the downsides.
• 18:31
  SPEAKER_01: This is what we're expecting for the implementation timeframe.
• 18:35
  SPEAKER_01: This is how we expect it's going to negatively affect our process for a while.
• 18:40
  SPEAKER_01: These things are all important to understand so you can have a proper viewpoint on what this change might look like.
• 18:49
  SPEAKER_01: Because right now, there are a lot of teams that have just dove right in, or dove in?
• 18:56
  SPEAKER_01: I don't think that's the right word.
• 18:57
  SPEAKER_01: They dived right in.
• 18:58
  SPEAKER_01: There you go.
• 18:59
  SPEAKER_01: Um, they dive right into AI.
• 19:02
  SPEAKER_01: And they they put AI in the, they told our teams, hey, make AI right while your code from now on.
• 19:06
  SPEAKER_01: I want you to be a project manager.
• 19:08
  SPEAKER_01: They kind of just dove right into the whole process without doing the fully evaluation.
• 19:15
  SPEAKER_01: And you know what?
• 19:16
  SPEAKER_01: They're finding out the negatives.
• 19:17
  SPEAKER_01: I'm not saying that AI is bad for your business, but you have to make sure that you fully evaluate it to say, hey, you know what?
• 19:27
  SPEAKER_01: Here's the downsides.
• 19:28
  SPEAKER_01: Because there are some significant downsides that, if you aren't prepared for, will really cause problems and get people frustrated.
• 19:36
  SPEAKER_01: And you might even lose a good thing because they didn't expect the downsides.
• 19:42
  SPEAKER_01: Where a good thing that did have more positives and negatives, but because those negatives were surprises, it really ends up tainting the whole process and you lose the good thing.
• 19:55
  SPEAKER_01: So make sure to be an honest evaluator of your options.
• 19:59
  SPEAKER_01: Don't just say it depends, but then not really be honest about the positives and negatives, the way to evaluate if it's if you should move or not.
• 20:09
  SPEAKER_01: Um, all these things are really important.
• 20:11
  SPEAKER_01: Okay.
• 20:12
  SPEAKER_01: So I would encourage you to make sure you cover all four categories.
• 20:15
  SPEAKER_01: Make sure you answer those questions and make sure that you understand your decision before you just make it.
• 20:22
  SPEAKER_01: All right.
• 20:22
  SPEAKER_01: Thanks for listening.
• 20:23
  SPEAKER_01: And as always, I am Tim Corey.
• 20:25
  SPEAKER_00: Thank you for joining us for this episode of Dev Questions.
• 20:28
  SPEAKER_00: When you're ready to learn to think and code like a professional developer, head over to IamtimCorey.com and enroll in a course.