Latest / You've Been Heard / 374- AI Automation Reality Check w/Jaron Jones

Transcript

374- AI Automation Reality Check w/Jaron Jones · You've Been Heard

• 0:09
  Speaker A: All right, we are live today, everybody. It's a very special day. We're talking with Jaron Jones. We want to talk about AI business adoption, best practices. What you think is the current landscape in the marketplace? Because there's some things that are going wrong, and I think there's a lot of places to make a lot of bad decisions. I'd like to hear from you what you think the current AI landscape looks like, what the reality is, and where we're making a lot of mistakes or what we're doing, or where we're making good decisions.
• 0:43
  Speaker B: Yeah, I think two really important things I'd like to touch on before we kind of get started on that. But Good, good context. One, MIT recently put out a study on companies adopting agentic AI. I think 95% of companies, it's either not working or they've. They basically scratched the project. There's a little more context, so I'd suggest reading it as with any paper, but obviously a lot of businesses are struggling with it and the landscape. I don't know, I. I think a lot of businesses, they obviously have problems and they have things they would like to automate, make less expensive, so on and so forth. And maybe those things have kind of been sitting on a shelf for a long time. And maybe you have like 10 or 12 really important enterprise workflows in your company, and maybe they all exist at certain levels of automation and abstraction. Right now with AI, somebody's basically coming into every company, multiple, multiple people, salespeople, people from other orgs, so on and so forth, and saying, hey, these 10 or 12 problems, AI can totally solve those problems. And maybe for one or two of them it can. For all 12, it probably can't. Maybe there's a specific company can, but probably, probably can't. And it's just, it's a landscape of promises above anything else. And I think that can be problematic. And the first place I start, personally, I think it'd be a good spot for us to start is really trying, trying to approach everything still as an automation problem and trying to look at those nodes. Maybe you have something that needs to call workers in your company and maybe they provide responses like 1, 2, 3, they get a digital input and it goes back to the database. And then some other workflows happen and somebody could come into your company, say, hey, AI can do all this for you. It's, you know, well, calling services already exist. They do it really well. They do it almost a hundred percent of the time. And, you know, automation scripts already exist probably Not a good work work example, but you know, you get another problem where you receive input from a customer and tech recognition or text recognition and you need to say, hey, we need to be able to tell how upset these people are. So maybe we can provide them with a different level of support, whether it be technical support, sales, something like that. You could have a person go through all those. You could try to use a service where you could use AI. AI is quite good at that, but just trying to almost. I don't know if anyone's ever played with like Lego Robotics or something where you have like five different things that are kind of reliant on each other, really trying to draw all those things out and look at what AI is actually replacing. And then you look at like the specific node or the specific part of the problem you've identified AI can replace, then actually start doing statistical tests or some modeling to see how well IT can actually replace that specific thing.
• 3:40
  Speaker C: At dissecting popular IT nerds, we expect to win, and we expect our IT directors to win. And one of those areas where we know that we can help you win is Internet Service providers. Providers. As an IT director tasked with managing Internet connectivity, few vendor relationships can prove more painfully frustrating than the one with your Internet service provider. The array of challenges seems never ending. Dealing with ISPs can try one's patience even on the best of days. So whether you are managing one location or a hundred locations, our back office support team and vendor partners are the best in the industry. And the best part about this is the none of this will ever cost you a dime. Due to the partnership and the sponsors that we have behind the scenes of dissecting popular IT nerds, let us show you how we can manage away the mediocrity and hit it out of the park. We start by mapping all of the available fiber routes. And we use our 1.2 billion in combined customer buying power in massive economy of scale to map all of your locations, to overcome construction fees, to use industry historical data. And we leverage aggregators and wholesale relationship to ensure you get the best possible pricing available in the marketplace. And on top of all of this, you get proactive network monitoring and proactive alerts so that you're not left calling 1-800-go-POUND SAND to enter in a ticket number and wonder, why is my Internet connection down? In short, we are the partner that you have always wanted who understands your needs, your frustrations and knows what you need when without you having to ask. This all starts with a value discovery call where we find out what you have, why you have it and what's on your roadmap. All you need to do is email Internet popularit.net and say I want help managing all of my Internet garbage. Please make my life easier and we'll get right on it for you. Have a wonderful day.
• 5:38
  Speaker A: Let's go back to that LEGO model because I do have LEGO robotics set and my kids play with it. So let's just go back to that metaphor a little bit and maybe break that down a little bit and, and compare it to any make believe manufacturing company maybe.
• 5:53
  Speaker B: Yeah, I mean really, it's just kind of breaking it down because the salesperson's going to come in and say AI can take care of this entire problem for you.
• 6:01
  Speaker A: And what do you think is the main problem that people are coming in and saying the AI is going to fix right now?
• 6:06
  Speaker B: Main problem? The AI is one fix.
• 6:08
  Speaker A: I mean it's like to me I'm just wondering what's what salespeople are coming in and selling with AI other than kind of just top level LLM type stuff?
• 6:19
  Speaker B: Yeah, I think agents and support, I think trying to specifically replace workflows. So somebody might come in and say hey, you can replace some of your IT support with AI. And I seeing some of our customers attempt to do this to you know, different levels of results and they say, hey, AI is going to identify what things it can take care of and what things it needs to send to a person maybe. And really that's trying to replace somebody who exists on top of a call queue, who's going to do the routing for company for like a call center metaphor. And it's a node. And the problem is you get a call from a customer, somebody has to identify or assess that call and then it goes through different levels of triage and support and they're saying hey, AI is going to replace this top level triage and might also replace some of the lower levels of support of things that it can take care of. That I would say that'd be a big one. I think people were definitely focusing on agentic stuff, but it depends on the provider. Like if you have a company that's scanning emails for vulnerabilities, that's not agentic. But those companies are saying that you're using AI privately, utilizing LLMs and deep learning at this point in time and trying to assess like whether an email is vulnerable or not. But it's hard. I want to say agents and I am rambling a little, but I think it is like very Specific in the space that you're talking about.
• 7:41
  Speaker A: Let me just make sure I hear you correctly, is the main point is for the sake of technology adoption, we should be focusing on automation first. And how does AI assist in that automation?
• 7:52
  Speaker B: Yeah, just, just looking at a problem, trying to contextualize the problem and trying to look at like each specific part of the problem and how decoupled, coupled it is from others, really diagramming it and trying to be more explicit about what AI is actually replacing. Because I think the conversation, like if you were to go to a conference and talk to someone is like, AI is going to replace this part of my business. It's like, well it's, it's not actually going to replace that entire part of your business, it's going to replace a part of that problem. So what part of that problem is it going to replace? And start talking about it is that level. Because you're not going to have a useful conversation if you're talking about AI replacing entire arms or business units. And I almost feel talking to some people, that is where the conversation's going maybe 10 or 20 years. But at this point in time, each service has its strengths and weaknesses. But yeah, contextualizing, identifying and I guess when you get down to that node and kind of like we talked about previously, like let's say you look at that call center triage problem of calls coming in, you know, instead of having a human, you have a AI agent trying to replace those calls. I really like confusion matrix. If you don't know what that is, I'd recommend Googling it. It's probably the number one tool that like a data scientist could give a stakeholder as far as evaluating how accurate model is. Because at the end of the day, like you want to have a conversation on how many true positives, how many false positives, how many true negatives, how many false negatives, how accurate is this tool? And those numbers are great and abstracted. But then you could take your human who's currently doing all those cloak use and routing and say, same thing, you know, evaluate anything against this and you could say, hey, this AI tool, we demoed it for this company for two weeks, we had way more false positives. That's going to be a problem for our business because we had this SLA with, you know, these five customers and, and maybe you could even go back to the AI company and say, hey, is there a way that we can only use it for these customers where it wouldn't interfere with the SLA or all these true false Positives as opposed to these other ones, but really just trying to get a more rigid framework. And to be honest, it's, it's really good, especially working with vendors because if they're not willing to give you a confusion matrix, it's probably not a vendor you want to be working with because every data scientist in that company knows how to pull one. And if they're not willing to give it to you, that's basically them not participating in due diligence from an AI perspective.
• 10:22
  Speaker A: So, so let's go back just to the beginning where, where some of the companies where you. There's a great article on MIT on why companies are pulling out whatever AI adoption, whatever, whatever area of AI that they adopted that failed on. Right. So first of all, do you have any examples that you can think of off the top of your head?
• 10:40
  Speaker B: Yeah, I, I have one company we work with in particular and I expect that their AI probably won't be there a whole lot longer. It's it similar to kind of the example I've been describing. They're trying to automate a certain part of their tech support process. Specifically AI is doing the routing for them. I think it attempts to solve some of the tickets.
• 11:00
  Speaker A: Okay, so more of like a should we open up a ticket, should we close a ticket, should we escalate a ticket, like that type of thing or.
• 11:08
  Speaker B: Yeah, yeah. And I think the problem this company is having specifically is, you know, it's, you know, it's not working as well as they'd like to. And their company in the IT space that relies on high SLAs, they rely on things working most of the time and their customers are saying, you know, maybe humans were taking care of this 95% of the time. And the customer satisfaction I get back from a human was, you know, let's say seven out of 10 on average. And the AI is just probably going to be returning lower results.
• 11:41
  Speaker A: So now let's layer in the confusion matrix. Let's from the data scientist standpoint or from it and, and to break it down from a very, keep it simple, stupid, from a top down C level, a business executive trying to make a decision. You come to the table and say, hey, let's take a look at this confusion matrix. First of all, define what it is and the, the matrix and how are we now going to decide? Go or no go on this, Pull this AI, it sucks, get rid of it. It's killing us or it's okay or the return on investment is good. I'm assuming you have a Confusion matrix is measuring all of these things.
• 12:18
  Speaker B: Yeah, I mean it's, it's measuring true positives, false positives, true negatives and false negatives. But that, that's important context. And one, you know that 95% failure. The reason a lot of projects fail is because the project should not unster in the first place. So we'll, we'll start there at the confusion matrix. You know this, if I'm saying this company, you know, 90% of our tickets weren't being handled correctly by human and 80% of them, or maybe 70, 60, 50 are being handled by correctly by AI. If they had done that R and D and just given a confusion matrix to management and said, hey, these are our current numbers. 90% of our calls are handled accurately, 10% are inaccurately. And, and the AI is only doing it. Maybe it's 88, maybe it's really close. But maybe the instance in which it's wrong, it's having a lot more like true positives as opposed to false negatives or something. They could say, don't spend time building this. We can't have this type of error where it tells a customer something worked and it didn't. That's really detrimental to our business. Even though it's right a similar amount of time, it can't be wrong in this specific way. That's the difference really between like a false positive and false negative. False positives are really destructive for businesses and trust and you would have never pushed out in the first place. And the kind of second part of your question, let's say the model was live and let's say it wasn't working to the expectations. They hadn't done this prototyping R and D evaluation prior really coming in doing the same thing. Because you will be in a lot of business situations where there are people who just don't like computers. I hate to boil it down to that, but that just happens sometimes, especially in a CIO or a technical role. And you can provide them something to say whether it's working or not. And there are a lot of instances I've been in where somebody says this isn't working. And I can give them a confusion matrix and say, hey, my model is right 94% of the time. Your technician was only right 89% of the time. I get when the computer breaks, it's frustrating for you guys, it's a new type of something breaking. But this is still much more accurate than a person. There's always going to be errors, but so on and so forth. Because People definitely when something's new, if they don't like technology, they're going to over index on something breaking. But I don't know if a human is wrong 10 times out of 10,000 and a machine was wrong three times out of 10,000, you can let somebody focus on those three times and say this is crap, pull it down, we're getting rid of AI. Or you could provide them with a real deliverable and say I get three is an ideal. You know, we all wish our products worked 10,000 out of 10,000 times, but still better than a human, cheaper than a human.
• 15:03
  Speaker A: So numbers don't lie. People lie.
• 15:06
  Speaker B: Yeah. And it's, it's a good way of looking at it. And I, I really like the Matrix because it's not just a yes, no, it does drill into like the type in which the model didn't work is I, I've come back to it a few times. But false false positives are really destructive in tech where the machine says this is done correctly, this is routed correctly, whatever, and then it closes the ticket, but the person wasn't taken care of. Am models have a habit of having a lot more false positives than human models. And those are the ones that really deteriorate, like business trust. Because somebody's like, I hate this new thing. It closed my ticket. But it's not taken care of. Whatever. Where usually models and humans are pretty similar in like the amount of false negatives they have where they think something's not an issue. So they kind of. Humans, humans mess up, but humans usually don't say something's done when it's not done. They're, they're definitely better about that. And they, that's, that's why I like the Matrix. It's not, it's not. Yes, no, it's like I said, it's evaluating the specific, how it's failing. And you know, if you're a business executive and you understand your business context, you can understand how destructive that, that type of failure could be. Or you could understand maybe this isn't a big deal. Maybe the agent, human agent was correct 90% of the time, but the computer agent is correct 80% of the time, even though it has four times as many true positives. Like this isn't a workflow that really matters. It's way cheaper, it's faster, we can deal with it because it isn't always about better. It isn't always about proving the model's better. Your confusion matrix could say you're new. Automation slightly works all the Numbers could be slightly worse, but you have other things to contextualize, like price, SLA, things like that. If you go to your CEO and you can say, this is 11% worse in this way, 5% worse in this way, 3% worse in this way, but it's four times cheaper. Sometimes that's acceptable, sometimes it's not. But it's good to actually make the decision with those, all those metrics in front of you. It's not just good to say, like, vaguely like, this is cheaper and better because it's AI.
• 17:13
  Speaker A: So I really like this because it gives. Well, first of all, it gives a very technical approach to decision making, but it also is providing a way to translate it into business language that makes sense for a C suite to make a decision on. So you're saying first you use the confusion matrix to decide a, is this better than a human, or is it not? Or what are the numbers? Basically, like, what are the numbers? First, is it even plausible? Is it even worth doing? If it is worth doing, okay, how much does it cost?
• 17:50
  Speaker B: Exactly, Exactly.
• 17:51
  Speaker A: And how many hours is it going to take to deploy? Then there's implementation, Then there's. I like this idea of how to not. This is what I wrote down. How to never start a project that shouldn't have been started ever again in your life, but that applies to so many other things in life and there's so many variables to that. You have, number one, this, this, this confusion matrix thing. But how would you even apply that to, say, vendors that you even knock on, on your door to begin with?
• 18:20
  Speaker B: I just worked with a vendor who's providing a technology for our business right now, and I'm not going to get too into the specifics of what they're offering, but the first thing I asked them was for confusion matrix, and they gave me one.
• 18:33
  Speaker A: Did he say, hold on, what the heck is that? Let me go back and get you one? Or he's like, oh, yeah, it's right here.
• 18:37
  Speaker B: There was a technical person on the call. Some sales engineers are really technical. Some are.
• 18:42
  Speaker A: I think the sales engineer is the best job in the world because you don't ever have to actually be responsible for any one project. You actually get to have a lot of projects and. But at the end of the day, you're not ultimately the guy implementing it. So it's like really cool. You get to talk technical all day and do all the fun work. You don't have to do any of the selling because the sales guy does all that part. You just get to talk Technical all day. It's actually like a pretty sweet job.
• 19:06
  Speaker B: Yeah. But I think they gave us 60% is a. Is a number for the true policies. Like, the model is wrong. More or less. 30. 30 to 40% of the time, depending on how you want to look at the true negatives. But for this business use case and what it was going to be automating, what is going to be taking off the table for us, that was acceptable. And I think that helped us as well because it helped establish a baseline and a metric and a target up front of this product's not going to work all the time, but it's going to work more than half the time, and it's going to remove a lot of.
• 19:43
  Speaker A: Just out of curiosity, was the things that it was wrong on. Are these, like, weird outlier type things.
• 19:48
  Speaker B: Or using vision recognition and actually think 60% is kind of an acceptable benchmark in that space right now? I wouldn't say they're outliers, but I would say, like, vision recognition is different ll in that it's kind of trying to categorize things. And I think certain things it's very good at categorizing, but certain things it's consistently not good. So I wouldn't say it's outlier, but I would say there are things that it's consistently not good at categorizing. Like, a really good example is if you had, like, a vision model that was supposed to categorize cats and he had a bunch of dogs run through and dog, dog, dog, dog. If somebody had a breed of dog that was like black and £9 or £10 and it ran across would show up as cat. Yeah, my. It would pretty consistently show up as cat. And I kind of see some of that in this. Like, it. It's failing a lot of the same problems. I kind of know why it's failing them because it looks like things that it shouldn't be failing, but it's not, you know, categorizing and it's falling in the wrong bucket.
• 20:49
  Speaker A: Can it learn? So it's supposed to be a learning. Is it. Can it learn? Can we, like, implement? Like, hey, by the way, that that was a cat.
• 20:57
  Speaker B: I think it has reinforcement learning. This customer, they have an option in the product where you can, like, flag. Like, the prediction is inaccurate, accurate. And my hope and how I would design a product is that flag would go back to the model trend, and it would kind of go back to when you want to make a model, you have to categorize your or classify your Training data. I would hope that us flagging it would go back and reclassify that training data. So like, let's say like a yield sign showed up and the model says the stop sign. I would hope that it would go back and fix that label in the training data, but I don't. It might just be there to make us happy. I don't know their internals.
• 21:40
  Speaker A: So the sales engineer told you, literally right to your face, it's wrong 30 to 40% of the time.
• 21:47
  Speaker B: Yeah. I think honestly for vision recognition right now, I think that's. I think there's definitely some people who would want to come flex and give higher numbers, but if people are going to be realistic about what they're seeing in businesses right now, I think that's for, you know, situations where you have a lot going on and you don't have like clear colors or barriers or kind of dividers between your target and some of the stuff behind it around it. From what I've seen, like Kaggle and other data science competitions, like what we're getting is an acceptable number. And I was happy that they were honest in giving that to me. And, you know, that also shows me that it's not BS and if I wanted to test whether it's BS2, I could just get someone in Tier 1 or Tier 2 and say, hey, we're going to demo this for two weeks. I could pull down 200 videos and I could have him manually do the matrix for me. And I do that a lot because I just don't trust the vendor and say, out of these 200, it categorized all 200 of these. Can you give me true positives, the ones that were right, true negatives, you know, the ones that were obviously wrong? False. False. You could just have a human do that and test these vendors too. And I would advise most people like, it's. It's great. I believe this vendor gave us a good number. But like also a confusion matrix is 4 numbers. I could ask ChatGPT what good numbers were and I could give it to you right now. So trust but verify.
• 23:11
  Speaker A: Yes. So number one, ask your AI vendor for a confusion matrix. If it's a specific, obviously, if it's a specific use case. What do you say about just end users using just LLMs in general and all of that?
• 23:23
  Speaker B: What do I say to it?
• 23:25
  Speaker A: Yeah, I mean, what do you think about, I mean, leaking data, company data, company sensitive data, all this type of stuff.
• 23:31
  Speaker B: It's actually kind of interesting topic because there's definitely a few approaches to it. Right now one is you can just block a lot of that. I mean they have IPs, they're better companies. You can just kind of go on from a security perspective and tell nobody in your org they're going to use it. And I have friends who work in like finance and government. That's kind of the case for them right now. Any LLM at all, just don't, don't or I think some of those companies are like chat and Gemini and stuff are developing internal contracts. These companies were, they have agreements on what they do and don't do. The other end of that is there are some browser extensions that are coming out. It's a tricky space because I think they do what they do well. I've, I've seen some of the source code for some of the startups. I don't think it's a terribly complicated problem. Same thing, you know, it's text recognition. You're going to have false negatives, false positives, there's an error right there. Even if the extension works 99% of the time, some small amount of data is going to get through and some of these companies are new enough. If you're a company, you know, who requires SOC 2 and like due diligence questionnaires from your vendors and stuff like that. Like some of these companies name browser extensions for the LLMs. I don't think they're at the point where they can work with enterprise level companies for that, that type of stuff. They're definitely more in startup mode. I think that's probably going to change rapidly, to be honest. I think there's a need, I think you're asking me about it because there's a need for it, but I think that's where most of the space is right now. Either like ban it, work with a company, get an internal model that's safe or Chinese browser extensions and find a relatively vetted company if you can.
• 25:11
  Speaker A: Or there's the extreme example of you're fired if you don't use it.
• 25:15
  Speaker B: Well, I'm talking about using it securely as opposed to using it in adoption. I think it's kind of like a Ferrari and that if you like, let's say everyone in your company is driving a Corolla and they have relatively the same tools and everyone can kind of drive a Corolla. It's easy to shift, you can get in, turn it off, you rent a Corolla. Everyone here could probably figure out how to drive a Corolla. Some people, you give them the Ferrari and they know how to drive. They're really, really good like developers or data analysts or things like that. And they're using it to like offload some of their work and automate. They're using it like as a second source of truth. I like using it for, for pull requests. Sometimes I just copy and paste the thing. I review it myself for a minute or two and then I kind of compare notes. I really like it for that.
• 26:01
  Speaker A: You walk me through that because that sounds like, like maybe how do you, how do you do that?
• 26:06
  Speaker B: Control A, control C, control V injur.
• 26:09
  Speaker A: Again you're talking, you're talking to a very, a simpleton from, from a data scientist perspective and math perspective. You're using things in a way that the normal human doesn't, doesn't normally do. So everyone else is probably laughing at me right now and they're like Phil, why do you ask that? But I mean, what are some of the things that use AI? How do you use it? Are there any tricks for the trade?
• 26:29
  Speaker B: I guess the problem was that don't require context. It's really good. I, I would say that's the biggest one for me. I guess to go back to the Ferrari example, I'd like to finish that because I really like it. A lot of the really strong developers, they just use it to offload work. They understand the context windows. It's good and not good. So to answer your second question there, I think it's really good for problems don't require as much context. So like if I had a data analysis problem or script that was confined to like one Python file or maybe like the libraries or dependencies are all in like one pretty tight repo, pretty easy to kind of get it in there. It does a good job of evaluating everything. Like everything's kind of like within bounds things that it struggles with. I do a decent amount of work in the database with like ETL pipelines and things, things like that. That can be harder for. Because it requires so much context for it to understand like all of your company's ETL pipelines, like what tables have what indexes, what constraints. It's definitely harder. So problem is less dependencies really. I'm saying context, but I think what I'm saying quite literally is dependencies. I would say it's a big one and the developers I see who struggle with it are the ones who doesn't understand everything. I kind of just said who. There's a lot of out of chat GPT and put it in and they're, you know, like someone who can't drive a Ferrari. They're better off in a Corolla. They're probably going to crash. I see a lot of developers who aren't as strong. I think the LL is making them a worse developer. I fear for a lot of junior developers who lean too much on the LLM because I think they're just in a Ferrari and I think they're crashing over and over and over again every day. And some of them will learn to drive the Ferrari and some of them won't. It's definitely like a deep end of the pool approach to.
• 28:19
  Speaker A: That's. To the, that's the glass half full, glass half empty. Like, is this going to be the dumbing down of society?
• 28:25
  Speaker B: Some people will use it and they'll use it to. As a second source of truth. Some people use it to check their work. Some people, I think who I work with a guy who has probably only been writing a whole lot of t. SQL for two years. He learned so much from ChatGPT and I know that there's not a lot of people in his department who's mentoring specifically because he's a really curious guy. And when he doesn't know why something's doing what it does or how to improve something, he is a big like what, where, when, why guy. And those people, I think can benefit a lot from LLMs, the just make it work people. I fear for them.
• 29:08
  Speaker A: The curiosity is one of our values. Like on the podcast, if I, if I was to ask every single person that was ever on the show, which is like 390 now, what if you were interviewing someone, you're looking to bring someone on your team? Like what is the. One of, what is one of the most important character sets? Or it's not even a skill set, or what's one of the most important values that they would have that they say curiosity, it's big.
• 29:34
  Speaker B: You know what my number one is? I've had a lot of time to think about this. Prepping students at the university where I teach. And then I, I like to be involved in interviews here because I think staffing is really important for culture, culture building. Asking people questions they don't know. And specifically I like to look at someone's resume and find a topic that they're relatively knowledgeable on, but give them something a little over their head. And some people have intensity to reach and just talk a lot and this and this and this. And they're like, oh, it's the tip of my tongue. It's the tip of your tongue for A reason. Like, you know, I know that you're close to knowing this, but I know that you don't know it. And some people are really good at saying I don't know, hey, I know this part of the problem, but I don't know anything beyond that. And just very little, much better at kind of contextualizing where they are and where they are.
• 30:30
  Speaker A: You're saying that's a demonstration of integrity and character.
• 30:34
  Speaker B: Demonstration of a lot of things. A lot of it's a demonstration of how quickly they're gonna be handle learning a lot of new things in a workplace too. Like if you ever get your follow.
• 30:48
  Speaker A: Up question, so what's your follow up question? So what would you do? Like what would you do to learn it? What's your follow up question?
• 30:55
  Speaker B: I'm definitely kind of a go with the flow person. I think people usually give you enough to come up with a good follow up question. If I had to come up with something kind of general to do after that, like if somebody is kind of rambling for a while, I'm probably not going to ask them a follow up question. I just don't think those are useful conversations. A lot of time if somebody said, hey, I know this part of the problem, but I don't know this part of the problem, I kind of start trying to drill into like I don't know, kind of where is this person at? I like to ask people like if I pay D for two weeks to learn more about anything, what would you learn about then? You know, why would you learn about that and try and just find where their, where their head's at is? And I think this kind of ties into AI and why there's tough decision making in AI right now. Every 10 years that goes by there's more layers of abstraction and technology. So like AI is kind of a new one. So you have like AI programming data and then executive. And I think that's why it's really hard right now because it used to just be programming data executive and there's this like new layer on top of everything. And I think same thing with developer, trying to figure out or you know, analyst or whatever are these big fixer people or these people like to focus more on solutions, architecture. These like nitty gritty people that want to like get in grind like binary trees or like do like rust performance optimizations and stuff like that. And knowing that can definitely help, you know, make sure that they end up being hired to solve the right problems in your company. Because you like get somebody who wants to do like rust optimization for example. And you hire them and you say, hey, can you clean up our Azure environment and you know, clean up our DevOps and this and this? Probably not a good fit. So just kind of trying to figure out like where people are at, what problems they like to solve, things like that.
• 32:49
  Speaker A: It blows me away how far we've come in technology and how many aspects in different. Just like what you just said, like whatever categories and subcategories of jobs and things that we would even need to hire someone for.
• 33:04
  Speaker B: Yeah, I always, I really like the term layers of abstraction because I think it really helps contextualize. Like you know, it's okay, you don't know this thing. It's like three layers from what your job is right now. Like if you're an executive, how far is AI actually from your job? Like there's the data part, there's the programming part, there's the AI part. There's, there's a lot going on there and there's more going on than there was 10, 20 years ago because a AI use building balls in the app. So you're expected, like I said, to understand the data, the AI and the app 10 years ago. You kind of just had to understand like the data, your business problem and the app. There's just more to worry about. And if you don't understand all those things, like at a relative level, you're probably just going to make more mistakes or you can find people you trust who do understand those things and get help. And that's the best advice I would give anyone is if you're not that type of person, it's not going to get less complex, just separation of concerns. Find somebody who does like working on those problems because like AI, AI is just so rooted in math and you know, I got in data science because I liked math. There are things I don't like working with and I'll do them if I'm asked to do them.
• 34:21
  Speaker A: What do you hate working with?
• 34:22
  Speaker B: I gotta know what I hate working on.
• 34:25
  Speaker A: Creative writing.
• 34:26
  Speaker B: I kind of a sicko. As long as something like is challenging, I like working on it. I really like puzzles. I really like, like intricate, complicated, like board games and stuff like that.
• 34:37
  Speaker A: Isn't that like non human? Anyways, isn't that the whole purpose of AI anyways?
• 34:42
  Speaker B: It is, it is.
• 34:43
  Speaker A: Like that one company that had like whatever 2000 people in India doing that one automated call it AI. Yeah, that was because it was cheaper, it made more sense. That was like a business case. So many things I Want to ask you now. First of all, the, the, the asking people questions they don't know is awesome. How would you know that they didn't know something, by the way? Just.
• 35:07
  Speaker B: Like, let's say I get a resume and on their resume, like they say that they are, you know, relatively good at like JavaScript, C and SQL. And I'll be like, oh, I, I actually know quite a bit of SQL. And I can tell by the things they listed roughly where they're at. And then I could ask like my feeler questions and I could say, hey, can you explain the different types of joints to me? And based on how they do on that, I kind of know what the next question needs to be. And then once I start to feel a little struggle, I'll kind of try and go like one step above and see what, see what happens. I think that's good too. This is going to sound weird, but I think it's good to see people in a point point where they're a little stressed or challenged because you gotta interview someone you're gonna work with on good days and bad days. And there, when you have a coworker, is a problem they can't solve. Some people are good to work with, some people aren't. And you can test that a little bit in the interview. That's always a nice thing.
• 36:08
  Speaker A: I know this sounds funny, but I interviewed this guy years ago when I was a store manager at Starbucks. This is like 25 years ago. And we were a very, very busy store. Like sometimes easily like 700 people in and out the door. Like a day lying out the door, like crazy busy. And it used to get really, really stressful. And if like the deployment scenario of like people in the cash register and people making coffee, if that, like, if that was off at all, it would completely, you know, just everything would fall apart and like, people would get like, really stressed out. So we did behavioral interviewing and I remember I asked this guy that had he just finished his second tour in the Marines? And I was like, I was just asking all those fewer questions, like just, you know, please tell me a story about the time you're like under pressure and like, why was it so like, whatever. And can you just describe everything that was going on and like, how you dealt with it? He was like, yeah, I was on front line in the invasion of Baghdad and I was driving the Humvee. It's like in. My tire blew out and I had to change a tire with all of us with like night vision goggles on and like missiles and, like, bullets, you know, things fine. I was like, forget about dude. I was like, you handle coffee. It was. I don't think I ever saw him, like, ever. Not in this really weird, calm, like, mode, even under the, like, most stressful situations of a stupid coffee shop. I want to ask you what. What is the end game for people in it? I'm just curious, like, what. What's life all about? Or, like, what's the end game for someone like you? I'm just curious because I ask everybody, and most people don't have an answer for it. Is it like, oh, I'm just gonna, like, cash out a 401k? Like, do you have any big dreams?
• 37:47
  Speaker B: I want to keep solving heart problems. My wife has probably six or seven years of school, and I think that starting a family while she's, you know, doing 70 hours of medical school is going to be tough. So I just kind of want to focus on solving hard problems, working with good people for that period of time, and then kind of as we get to the point where we can start a family, I think we're kind of both have to evaluate. And I think it's important to, you know, say now that, like, one of the best lessons I've learned is don't make decisions for yourself in the future. It's easy for me to say now. Like, I know where I'm going to be in seven years, and I want to start family, and maybe I'll want to do this, this, this, and this work. I know what I want to do for the next five years. I know what I'd probably want to do until she's out of school. But, like, I'm not going to make decisions for future me right now.
• 38:38
  Speaker A: Yeah, no one ever knows where they're.
• 38:40
  Speaker B: Going to end up.
• 38:41
  Speaker A: What are some of the smartest minds out there right now? Like, are just some of the smartest ones, because I have friends that are PhD, like, data scientist guys that, like, went in and sat in on interviews at, like, Facebook years ago, and they put them in a room for, like, 24 hours, and they're like, all right, here's a problem. How do you solve it? And then, like, they'd come in and see what his solution was, and they're like, okay, and not good enough. Like, you don't get the job, someone else does. I mean, these are, like, some pretty smart people. You having any finger? Any finger on that?
• 39:10
  Speaker B: Yeah, occupationally. Because I think that's kind of the question. I think I keep going back to solving problems. I think People who are solving hard problems. Like I think a really example and a really eye opening thing for me is I have a longtime friend who lives in New York right now and he has a partner who's works at a trading desk for a pretty prominent bank. And they didn't study business finance, they didn't studied physics at Harvard and they kind of touched on that. And I kind of, I went for a run, I kind of run our bike every day. And that gives me tend to think about things. And I was like, what does that really mean? Why did this company that's probably much better at assessing talent than you, I or probably anyone who's gonna listen to this. Why are they identifying these people and going after these people who don't have an occupational match to, to this job and someone who studies physics and undergrad at a good university is probably better at solving hard problems than most of us. And I think that's what it comes down to is can you solve really hard problems and then do you hopefully have some tools to solve those problems in the modern age? And a lot of that is math. I mean data science and physics have a lot of similarities there and that you have a lot of math and you can kind of get in, you can do ETL work, you can do programming work, you can do data science, you can do, you know, machine learning, AI, whatever, as long as you have that background in math. If you don't have that background in math, you're kind of hoping that you memorize keywords and understand some technologies. But under the hood, like I said, ML and AI is statistical in nature. Like data science really as a discipline is very, very, very statistical. And if you don't have that backbone, it's kind of like AI is a car and all that stuff is the engine. And maybe you can learn a lot about the tires and the paint job and all this other stuff, but like the engine, to really understand that it's, it just requires a certain level of math and statistics. And I, I think the smartest people are the people that are working on those problems. And if I was someone, like if I went back to college right now and my goal was to future proof myself, I would study math or physics.
• 41:28
  Speaker A: My dad was a math major, went to med school.
• 41:31
  Speaker B: I, I have actually heard that's a very good idea and I'm actually seeing it now with my wife. Because you have to learn like relearn biochem and micro and all that other stuff and meds, like you have to relearn everything. So why not learn something cool and interesting and undergrad and just give yourself more context?
• 41:50
  Speaker A: Yeah, he did. He was a math major. He was gonna, he was gonna be going to some kind of engineering, and then he was like, I can't do that. I just, I'm not excited about that. So he actually got into medical school his third year of Bowdoin College. And then he didn't even graduate college. He just went straight to med school. They actually took him into medical school before he graduated. So he was like a guy that went to med school without a college degree. So what do us other laymen do that are not math geniuses? What's the second smartest people in the world? What, what should we study?
• 42:22
  Speaker B: I would say get the baseline so that you, like, have the tools for dialogue. So let's say you're sales engineering, wanted to work on software. They'd probably teach you. Like you need to learn about APIs. This, that, this, this and this. To talk to these people.
• 42:36
  Speaker A: Memorize a thousand acronyms and yeah. Know what an IP address is? Just don't touch the router.
• 42:44
  Speaker B: Data science, kind of same thing. Like, if you really want to be able, like if you're working with a vendor or a startup or you know, your own internal company, you want to suss out whether you're maybe a CEO is listening to this and he wants to be able suss out whether his CIO is bullshitting him or not. And he must find some better language to do that. Like we talked about Confusion matrix. That's a big one. Confidence intervals, like, be able to say, like you have a certain subset of data and then you have certain amount of like maybe negatives and positives. Like, can you say with what level of confidence whether you believe this thing or not? Because everything exists on the spectrum of 0 to 100. But what does that mean when. When you're working in the research world, you know, someone can tell you, this is 99, this is 95, this isn't 90. Like, like being able to contextualize those incredibly important because you're never a hundred percent sure, even when they're working in like vaccine research or stuff like that.
• 43:37
  Speaker A: You mentioned the vaccine number. Well, even those 99% numbers weren't those. Those can be played with. The numbers can't even be played with as well. It's like an SLA. It's like you have a 99.9999% SLA. What does that really mean? Yeah, that means hours of downtime over thousands of customers over a year. Over a massive geographical area or use.
• 43:57
  Speaker B: Big Microsoft, not report the outage. It doesn't reflect your sli.
• 44:02
  Speaker A: You know, like, who cares? It doesn't mean anything.
• 44:06
  Speaker B: What other stuff is controlled? And like I actually have some publications in that space and my wife does too. And generally stuff there is controlled, embedded within reason. I'm not going to get too into that, but there are some fields of science and research where you can just put confidence under a ball on a paper and somebody's probably not going to try and recreate the study or experiment. Generally when you're in the space where something's going to go into human beings, somebody is recreating that and validating it. So there, there's sound assurance there.
• 44:35
  Speaker A: But last question, you mentioned last time, some, you just had some good suggestions on like hey, go take a udemy course or something like that. If you're IT director, mid market, IT director, whatever, Fortune 5000, you might not have a certain level of knowledge to be able to break things down. What, what would your suggestions be?
• 44:57
  Speaker B: My, my biggest suggestion, I actually kind of just came up with it from our last 10 minutes of discussion. But I feel very good about this is fine.
• 45:07
  Speaker A: How, how, how good do you feel about it? Are you like 95% good?
• 45:11
  Speaker B: About 99 to 100. I, I feel very find courses online was just like Introduction to Math for Data Science or Introduction to Data Science. And you don't have to go deep. Just go like maybe two to five hours in, learn enough jar and learn the baseline. You know, I did Coursera course years ago and I was kind of getting into it and the first unit of the course, they kind of covered all the key stuff and then the rest of the units that got into it like deep. You don't need to get into it deep if you're not a data scientist. But you want to learn the key stuff. And the piece of advice I want to give very explicitly is find stuff online for data science, not AI. I think a lot of the AI stuff out there is really convoluted. I think it's really product focused on like what version of chat GPT is better than the other one.
• 46:00
  Speaker A: Like hypey, hypey type of minutia stuff.
• 46:04
  Speaker B: Yeah, find something as do says do science, not AI because it's probably going to focus on the bones, the language and the communication more because it's less centered in hype and it's more centered around the discipline in which all this is built. To, to be frank, it sounds like.
• 46:20
  Speaker A: If you weren't Too busy. You could write that course yourself just to get something on.
• 46:24
  Speaker B: I think there's a lot of really good stuff out there, to be honest. Andrew Ng, he's. I did his courses year years ago. I think he's was mostly in course here at the time. Hisa was great back then too. I was kind of making decision around like 2019 or 2020 after we moved, whether I wanted to do data science or quantum computing and his courses and seeing some of the problems that he I could solve and getting to actually work with the math and the actually having the calculus and stuff like that kind of deeper on the course which, like I said, you don't have to get into. That got me really excited and that's one of the reasons I got into data science. I know he's still making stuff. I see him on my LinkedIn sometimes.
• 47:07
  Speaker A: I'm giving you the last word, final word. It's been a pleasure having you on the show. What's the most useful thing you'd have to say to all the IT leaders out there right now?
• 47:19
  Speaker B: Focus on data. I think models, a service will become more popular where you can kind of take your data and plug into a model. I think there will be new AI technologies this year and next year, five years from now, 10 years ago. Getting something that can deal with all of your company's data, whether it's like staff changing policy procedures, stuff like that, is just super important because there aren't a lot of services that can clean up your stuff and get it ready for AI and AI and machine learning and deep learning. All these tools, they don't work well with dirty data. And if you get good data, you could probably hire a consultant, have them come in 100 hours and build a great model for you because they'll say, this is good. I don't have to do any cleaning. This is ready for classification. But it's just kind of the idea of it's almost like data is lumber and today we're building ships and tomorrow we might be building houses and then 10 years from now we might be building apartment buildings. But we're always going to need lumber. So focus on your lumber.
• 48:24
  Speaker A: Jaron, very pleasure, man. Pleasure having you on the show. Thank you. Thank you again. Have a great day.
• 48:28
  Speaker B: Yeah, you too.