Latest / You've Been Heard / 427- The Hidden cost of AI w/Cody Aldinger

Transcript

427- The Hidden cost of AI w/Cody Aldinger · You've Been Heard

• 0:09
  Speaker A: Welcome back to another you've Been heard the space where we've built specifically for the IT leaders of today and the rising stars looking to secure their own seat at the leadership table. Our goal is to ditch the corporate scripts and have the kind of raw conversational deep dives it actually takes to lead an organization into tomorrow. Cody, we really appreciate you joining us today. Thank you for your time and for being so open to sharing your experiences with our community. Over at you've been heard.com to our listeners. We're sitting down today with Cody Aldinger, the VP of IT at KLJ. Cody manages the technology landscape for 30 offices around the US reporting directly to the CEO. A real gem in Cody's leadership in his philosophy is his blunt realization that a huge part of his job isn't technology, it's sales. He spent his day convincing business leaders that IT is worth listening to, moving beyond just keeping the lights on to becoming a vital voice at the executive table. Cody, we got a lot of ground to cover from the gap between AI hype and business reality to what it means to be the sole technical voice on an executive team. But I'd love to start by handing the floor to you. Could you introduce yourself to our listeners and share a bit about your eight year journey to the vpc?
• 1:39
  Speaker B: Yeah. Awesome. Thank you Mike. Appreciate you having me. So yeah. So as you mentioned, Cody Aldinger, I've been with KLJ eight years. I've been in the IT space since 2004. Four really? So 22 years. Started as an intern in a different engineering firm. Worked my way through kind of the basic help desk support stuff that most IT people go through. So worked through help desk support and system administration, did some help desk leadership, kind of managing that support team. Was able to take a little bit of a kind of a side journey into software development, which I think is a kind of a unique opportunity. I guess it seems like in the IT space, once you pick a path down the development world or the infrastructure and support world, it's hard to get experience in both. So I was able to take two years and do some development work which I think really helped my kind of overall strategy when it comes to IT stuff. And then I moved on to a hospitality firm where we was part of a company that worked in and owned and operated 186 hotels throughout the U.S. okay, part of about a 5,000 person company. That company was then bought by another bigger company and then eventually moved over to KLJ where I'm at now.
• 2:59
  Speaker A: So let me jump in real quick there because I have a touch of hospitality tourism in my history too. Bartending, room service, all that kind of stuff. What were you doing at and cause you were already in your technology career, so were you doing software development for them or.
• 3:14
  Speaker B: So I moved over to them to kind of help start a bigger IT team. We were part of kind of a 250 person maybe corporate oversight of the hotel company. Right. So I was not in a physical hotel, but we were in the corporate office and I provided a lot of kind of the strategic leadership on the IT side of the house there as well. Started a small help desk team when I moved over. There was only I think three people at the time that were managing this. So needed a lot of work, helped to build up that team. And then I switched over into a role that was really focusing on integration and automation. So really moving things like data entry that happened at the hotel properties and got shipped back to the corporate office on paper and was hand entered moving that stuff into more automated processes. So I think 60 plus properties and
• 4:05
  Speaker A: they were still sending in the paper.
• 4:08
  Speaker B: Yeah, we were 186 properties and yeah, when I started there we had back rooms full of bankers boxes and they were three months behind on entering financial data from the properties at a team of people that was just doing data entry. Wow. So being able to take technology and find ways to automate that, get it in to the systems in real time, this is just kind of a game changer for what year was that?
• 4:35
  Speaker A: Because yeah, that sounds like it was
• 4:37
  Speaker B: the mid teens, 20, 2012. So now. Okay, early, not that long ago. Yeah, I was there from 2012 to 2018 so.
• 4:47
  Speaker A: Wow. Okay. For them to be still doing hand entry of stuff that was printed at another location versus integrating all of the systems and actually the hospitality tourism for me was where I learned that lesson. I kept telling myself, I'm never gonna do sales. I'm never gonna do sales as I'm waiting tables and upselling the special of the day or bartending and finding out what we had too much of in stock and trying to help move that. And that's when I had that aha moment of everything is sales. And so I kind of had that before I went into technology. What caused that moment for you?
• 5:29
  Speaker B: Yeah, I grew up at a young age in the food industry. So my grandpa and eventually my aunt owned a restaurant my whole time growing up. I spent five and a half years as a fry cook and customer service and my aunt had driven into me the customer Service mindset. And then when I got into it, support it kind of carried with me. I think I was fortunate to be able to realize that a early age that the people that I'm supporting on the help desk were my customer. And my goal was to make sure that they were happy and they enjoyed the interaction. And as I kind of worked through my career, through leading the help desk team, eventually to leading larger IT teams, that is still one of my major focuses. So I try to instill that customer service mindset and the sales piece of it kind of comes along for the ride a little bit. But I think as I began really in my career implementing new solutions, I started to realize that a lot of what I was doing is trying to convince people that the solution that I'm pitching is the right one or at least is a solution. Right. Working with them to understand if it's the right one. And a lot of times when it comes to technology, people don't understand what's possible. So I'm coming up with crazy ideas, explaining to them how it's going to make their life better. And once I convince them, then it's a matter of proving it right, Then it's a matter of kind of earning that trust and saying, hey, this is what we said we could do. And here's the deliverable that does that. So.
• 7:04
  Speaker A: And that customer service mindset, I wonder how much you attribute your success and your growth from help desk all the way up into the leadership to the customer service. It's another one of those core tenants. Yeah, everything sales. And for me, the only differentiator is customer service because there are plenty of people who can sit in my seat and do what I do. Plenty of companies that can do what our company does. And so somebody will always take the business or be there.
• 7:35
  Speaker B: Yeah, I think that speaks volumes to me just personally as well. I'll be the first to admit I'm not the smartest guy in the room. I'm not the most techie. I'm not going to be the guy that rattles off acronyms all day long. I really kind of thrive on being the middle guy. Right. The one that can take the tech talk and explain it in a way that people who aren't techie can understand and then vice versa. Right. Taking the business need and what people are looking for and trying to bridge that gap with technology. And I think that's really where I can connect with our end users or business leaders.
• 8:12
  Speaker A: All of the interviews and all of the guests that I've talked to on You've been heard that seems to be our core thread and our core capability is that piece being able to speak both languages, be able to talk to the techs and then be able to talk to the executives and, or just the regular person sitting at the, at the computer out on the floor doing whatever job they're doing. I think one of the other core tenants is our ability to listen and then. Solutioneer.
• 8:44
  Speaker B: Yeah. And I think when it comes to AI, right, bringing this kind of back around a little bit, that is 100% the number one thing right now and what I'm doing a lot of in my current role is listening, understanding what is the problem that the business is trying to solve and understanding if AI is the right tool for that job.
• 9:12
  Speaker C: Most IT leaders don't need more technology, they need less noise. Over the years, infrastructure quietly gets complicated. Multiple locations, multiple Internet providers, different contracts, separate bills and reactive support. Nobody plans it that way, it just happens. What we do is vendor neutral advisory. That means we, we don't sell platforms or force change. We start with low friction areas. Internet, telecom, Microsoft licensing. Places where complexity builds quietly and can usually be cleaned up without disruption or capex. Sometimes we find savings, sometimes we find nothing. Either way the outcome is the same. More clarity, less distraction and it that's boring in the best way possible. If you want a second set of eyes from people who sit on the buying side of enterprise infrastructure every day, just go to you've been heard.com and answer the questions to simplify, streamline and save.
• 10:07
  Speaker A: So okay, and I'm going to kind of just dive headfirst into the deep end of the pool on this one and I was going to bring us right back to AI too. So I'm glad you circled in on it. So AI and the expectations, the reality and the in practice, there are so many leaders in the hype cycle up at the top of that thing, at the highest peaks of the hype, thinking that this is going to be the ground or the it's just earth shattering, change the world kind of thing. And you're talking about okay, what's the reality of it? So what are you seeing in that disparity? How close is reality to the top of the hype curve or is there a large disparity?
• 10:54
  Speaker B: Yeah, I think in some ways it is, it is going to change the world. Right. And it already is in today and it's still in its infancy. With that said, I don't know that it's going to change the way people think it does that it's going to. Right. You hear all the horror stories of we're all going to be dead in three years because AI is going to take over the world. And I'm not worried about that. Right. That's the fear. I think right now our limitation is just our understanding of how. How best to use is going to be another tool in the toolbox. And quite honestly, I compare this, and I know a lot of people do, to the dot com boom and all that kind of stuff. It's like you don't get to pick whether it's coming or not. You just have to get on board with it and figure out how best to use it, how to protect against it, and how to use it to your advantage. And is there going to be bad things that come from it? Absolutely, there already is. But there's also a lot of good. And if you look back at when the Internet became a thing and if people said, I'm not doing that, it's scary. Where are they at today? So you don't get to pick. You just got to like everything with the good and the bad that comes with it. You got to dive in and learn it the best you can and understand how to protect yourself and how to use it to your advantage. So I get asked a lot in my position of is my job going to get eliminated? How many people are going to get cut because we're this AI tool that we're putting into place. That's not the goal for us anyway. But I don't think it's the goal for a lot of people. I think it's trying to make our lives better and trying to make the monotony go away and make everybody's work more efficient and more enjoyable.
• 12:27
  Speaker A: Yeah, I see and hear that part of it. I also hear, of course, the fears, those questions of, hey, is, is my job going to get eliminated? I'm trying to remember what conference I heard this at, but I think it's going to be more of. It's not going to be AI that's going to replace me. It's going to be somebody who uses AI better than me or uses AI, period, when I don't is going to replace me because they're going to get rid of some of that monotony and they're going to move a little bit faster and they're going to accomplish it a little bit more through that. I was just having a conversation with somebody else a little bit ago and we were talking, and I wonder what your thoughts are on this. The word hallucination and how we're using hallucination whenever the responses or when AI drifts. That's another term that I'm starting to hear. A lot of guardrails, hallucination and drift, the hallucinations. I think we're doing ourselves a disservice. I was actually just talking to one of my kids in between interviews and saying that I think utilizing hallucinations, we're not categorizing the problems, the common mistakes that are coming from the generative models and thus trying to fix that specific mistake and so that we can tighten up and get it better. We're just saying, oh, it's a hallucination. It just happens. Bots.
• 13:55
  Speaker B: Yeah, Yeah. I think you're spot on. I think the word hallucination is a fancy word for flat out wrong. Right. It's just wrong provided the wrong answer. And you hear a lot of comparisons of AI tools or like new, fresh out of college people that maybe don't have all the answers, but they're eager to provide an answer. Right. There's lots of different things and analogies that you hear, but quite honestly, I think it comes down to the person interacting with the AI tool to tell it exactly what you want. Right. If you have something in your mind that you're expecting to see as a result, lay that out as clearly as possible to whatever AI tool you're using and you're going to get better results and you're going to get less hallucinations, if you're clear and can communicate it with it properly. And I think that's probably the biggest challenge that I see right now is prompt engineering. Right. The term prompt engineering is becoming a thing, and it's becoming the center of a lot of training courses around how do you talk to these agents to get the best answer that you can. And that's a real thing.
• 15:01
  Speaker A: Yeah, it most definitely is. Again, another one of the common threads, I'm trying to remember the exact way this one was stated to me, but it was that we're moving away from being the content creators to being the context creators that we now have to really provide that context, which is that prompt engineering is providing the context of the solution that we're looking for. One of the simplest ways I can put it was somebody challenged me to take a transcript and to create a prompt for the cio, a prompt for the CEO and a prompt for the cfo, and feed the same transcript into all three of those different prompts, looking for the summary, and then recognize what's different amongst that. So that you get a more holistic answer from the summary versus just that narrow point of view. And it's a chance for us to expand some horizons that way around that. Prompt engineering.
• 16:04
  Speaker B: Yeah, I've not heard it put into that form before, but I think that's spot on. So I took a virtual training course here a year or so ago and that was the first time I'd seen a prompt template in something that was a page long from a prompt. And we're all used to Google searching right now where it's 10 words maximum and it's totally different now. And people just haven't figured that out yet in a lot of ways. So a lot of my job and my team's job as we roll out AI tools is going to be training people on how to use the tool properly. Because if we just hand it over and say here's a fancy tool that can do lots of cool things, use it. They're limited by their own creativity, they're limited by what they're used to. Google searching. And now it just becomes a different way to Google search. So we really have to train that context. And as you said, how do you create the context of what you're trying to get out of it?
• 17:01
  Speaker A: Yeah. And so then there's always the fun around like, well, so I too have to create some of the training and provide some of those trainings and starting to bring in things like the skills and the memories and then again the prompts and the context and then teaching people when to break into a new conversation. Yes, it's the same tool, but break into a new conversation because right now it's taking all of that prior conversation plus the new 10 words that you've added to ask this new question that's on a completely different topic. So unless you want to know about how the anti flea stuff that you're putting on your dogs affects the lawn, whatever you're feeding the lawn to grow the lawn. Because those are two separate conversations that you've had or questions that you've asked the generative model in the same conversation. Now it's trying to correlate those two things together versus okay, this is a distinct conversation. So let me start a new conversation and let's just talk about the dogs or let's start the new one about the lawn or how do I auto process my email so Mark. And hopefully people are recognizing how much it can help them with those little, the monotonous tasks. This is one way I tell myself I love computers because I'm kind of Lazy. I want the computer to do the work for me. So especially any of those repetitive, monotonous tasks. I hate those repetitive tasks.
• 18:36
  Speaker B: Yeah, it's funny.
• 18:37
  Speaker A: People love them, but go for it.
• 18:39
  Speaker B: I was just gonna say it's funny you say that. Cause I've used that my whole career as well. I get people that are asking me like, man, how'd you figure out to do this stuff? Or why are you so smart with automating this thing? I'm like, cause I don't want to do it. I'm lazy. I don't want to do it. So that is my sole focus. When some of that stuff comes up, I'm like, I don't want to do this anymore. I'm going to find a better way. And it's not so much that I want to learn the new cool way, it's the fact that I don't want to do it.
• 19:04
  Speaker A: Yeah, yeah. I'm copying and adding formulas to Excel spreadsheets and all of that kind of stuff. The Vlookup.
• 19:10
  Speaker B: So yeah, I think one of the biggest things that I've seen as a benefit and this has come up just kind of recently, I'll give a little plug for a book that I read that just was recommended to me by somebody else. But the AI Driven Leader is a book that I read here recently and it kind of, it really emphasized the use of AI as a thought partner in using that terminology. And that really has kind of changed the game for me and a couple others that I've been working on. Our business leadership team of help me think through this thing. Right. So you tell the AI agent I've got this problem. And one of the other tips that this book says is ask me one question at a time to get enough context before you provide a solution. Partnering those two things together has kind of shattered the glass ceiling for me and a couple others where you start to realize that AI tools are better than a Google search and they can really provide a very different train of thought and help you work through kind of out of the box solutions that you maybe wouldn't have thought of before. And one of the people that I'm working with right now, one of our other business leaders just said to me this morning, I'd never realized how much I needed a thought partner and it's game changing.
• 20:29
  Speaker A: I'll tell you which model, if you don't mind me asking.
• 20:33
  Speaker B: Yeah, I mean, we're pretty much in Microsoft Copilot right now and just using their default auto model selection tools. I've used it with ChatGPT and some of the other basic stuff as well. And it works very much the same. But we as a company are very Microsoft heavy. We're in Microsoft Teams and all the Microsoft Suite. We've got Power BI reports and using Microsoft's fabric environment for data modeling and that kind of stuff. So Copilot made a lot of sense for us because it's got access to the entire Microsoft graph. So that's been our tool of choice so far. We're kind of evaluating whether that's still the right long term play. But I think for right now it is.
• 21:14
  Speaker A: I'm interested because I started down that path with our organization and then I actually quickly jumped ship. What do you think about the agency and has it changed? Because the last time I was really working with Copilot, I would ask it to do something and it would come back and say, oh, great idea, but I cannot interact with this. I can't start a Word document for you. I can give you the answer here and you can copy and paste it. Yeah. But I can't generate it for you.
• 21:45
  Speaker B: It's getting better. I'll say that. It's definitely not the best AI tool on the market. Right. And I think there's reasons for that that we can dive into or not. But I think we could potentially get better solutions at times using another competitor. You look at ChatGPT as an example. They do one thing and they do one thing really well.
• 22:07
  Speaker A: Right.
• 22:07
  Speaker B: Copilot is a part of Microsoft and Microsoft does a lot of things and there may be Microsoft, my experience has had a kind of a reputation of kind of starting slow and then all of a sudden they catch up and they're best in breed down the road. So we're maybe.
• 22:21
  Speaker A: Right. I don't buy it.
• 22:23
  Speaker B: What's that?
• 22:24
  Speaker A: I said that. Or they buy it.
• 22:26
  Speaker B: Yeah.
• 22:26
  Speaker A: Ride best to breed and incorporate it.
• 22:28
  Speaker B: Yeah, exactly. So for us we're maybe riding that out a little bit, but I think there is enough better benefit to us from a security standpoint and from an integration standpoint that I don't have to spin up an MSP server to get ChatGPT to integrate and then say, well, it integrates with these five things in Microsoft, but not everything. Copilot's kind of there and out of the box. It's the easy button, if you will.
• 22:54
  Speaker A: Yeah. Okay. So connectivity and access to the full Microsoft graph and all of the endpoints does make sense.
• 23:03
  Speaker B: Yeah. I was just going to say to your question on the agency, I think it's getting better. There's still some things that it doesn't do well. I don't think it creates PowerPoints very well. I can't tell it to delete all my emails for me. It'll say, I recommend you do this, but it's not going to click the buttons for me. Claude has its desktop app which can do some of the button clicking. I think they're getting there, but they're not quite there yet. So.
• 23:28
  Speaker A: Yeah. And actually that one leads into another area. The security of it and the use of other tools and the public tools and the shadow AI. While team members or peers, co workers are waiting for us to roll this out, waiting for them to go through, we're waiting for them to get through the classes so that we know that they're using it responsibly. They're waiting for us to make them sign whatever documents so they can use it locally and. Or they're not waiting at all and just going to Gemini and GPT and Plot and pick a. Pick whichever one. Nano. Are you guys doing anything in that realm? What are you guys?
• 24:08
  Speaker B: Yeah, and I think for us that's what's held us back a little bit from being further along on the AI journey than what we are today. Because we wanted to get the security in place first. If I go out there and start training people on here's all the cool things you can do with AI tools, but we don't have the security in place, it's going to take off like a wildfire and it's going to be really hard to wrangle in at that point because once people get on a certain tool of choice, it's going to be tough to get them to switch. Right. Especially if I like Chat GPT better. Copilot doesn't give me as good answers. Yeah, I get that. But there's other benefits with Copilot for us anyway. So having those conversations is going to be a lot easier if we can limit it from the start. So we have been going through the security process of it first. We're about four weeks out from having Microsoft Purview in place and Purview is going to do that security for us where it will limit AI tool usage unless otherwise approved. So from your work computer, if you try to go out to whatever name a system, it's going to deny it. Right. It's going to block it and say, you can't use this. It's blocked by organization. They would have to come and request access, which then allows us the ability to Kind of vet out what that tool is and its security and all that. Once we get that security posture set up, then I'm more comfortable going out and saying, hey, we're giving you Copilot to use. Here's how you can use it, here's some of the cool things you can do with it and kind of let them go at that point.
• 25:42
  Speaker A: So how much development, how much advanced usage are you and your team or how much are you enabling that today? So you know, the code creation cowork is one of those platforms for that, but Copilot tries to do some of that.
• 26:02
  Speaker B: Yeah, good question. So we're not doing a whole lot of cowork yet. We've got a small use case. We've got about 650 employees in the company. I've got 39. I think right now with Copilot Premium licensing, everybody else has the Copilot free chat. So obviously the difference, the Copilot free chat is just web based models. Copilot Premium has access to the internal graph. You can create your own custom agents. I have one of our business leaders that I worked with that had the probably the best use case so far and that's in proposal and content creation where they're using it to help analyze RFPs, help review and create content for proposal documents. And they've really seen a lot of benefit from that. They're helping it, using it to update resumes. So as people work on many different jobs a year, keeping those resumes up to date is a challenge in itself. So they're using it for things like that as well. So that's probably the biggest use case of what I would call custom co pilot agents right now. And I think we've got a couple other people that are starting to use it for research, feeding it tools and feeding it websites they typically would use for research, kind of centralizing that as well. So those are probably the biggest two right now. So nothing too automated yet. And I think everybody's skeptical and hesitant as myself going back to the hallucination thing, you don't want to just pump something out the door automatically with AI tools right now.
• 27:37
  Speaker A: Yeah, yeah, you definitely want to be or wise consumers of these products are going to be validating everything before they let it go. I've seen a couple of things generated and people just attaching it, throwing it out there without really reviewing it. And one, it's kind of obvious and two, you start to find some of those hallucinations and it takes away some of the validity of what's being produced or trust.
• 28:04
  Speaker B: Yeah. There's a lot of reputational risk that goes along with that as well. Right. There's legal risk, obviously, in some cases, and there's reputational risk that your customer is looking at you going, I paid you to do that, so I could
• 28:17
  Speaker A: have asked that made that prompt, too.
• 28:20
  Speaker B: Yep, exactly. We talk a lot about the time saved on content creation or doing the task is going to be spent on quality control before it goes out the
• 28:30
  Speaker A: door instead, or into the context to create it. Correct.
• 28:34
  Speaker B: Yep.
• 28:35
  Speaker A: What other challenges are you running into? What's an unexpected challenge that you've come across with this realm since starting on the journey?
• 28:46
  Speaker B: I. I think the go back to the glass ceiling thing, everybody's got kind of this glass ceiling of what they thought think AI is for and what it's capable of, and it's capable of so much more, but nobody knows how to use it yet. And that has been the biggest challenge for me as I'm working with people to break through that ceiling. Show them the aha moment. Right. Show them what is the use case that I can put in front of you and have you get amazing results that makes that light bulb go on. And it's different for every single person. So I have to do a lot of listening and asking questions of what's the monotonous stuff that's taken away from you, doing the productive stuff? What are the things that you spend most of your time doing and trying to really hone in on the use case? I've failed in that more often than I've succeeded. I think at this point where I think it's going to be really cool and I show them a way to use it and also, like, yeah, all right, whatever, that's useful, but it's not the aha moment that we're looking for. So I think for me, that's the challenge that I'm kind of battling right now is you look at for us, we have 650 employees. If we want to get good at it, how do I find 650 different individual aha moments and get that out there?
• 30:05
  Speaker A: Yeah. How do you make that epiphany spread? One of the ways that I'm hoping to approach that is really not not looking for the 650, but trying to find the individual, the hungry ones that are, like, trying to bull AI from us and say, hey, give me. I want to try. I want to play. And making them or helping them to, one, enjoy their aha. Two, to leverage the tool and show what could be and Using that as the attack vector for the rest of the people around them, the rest of the department, the rest of the facility, or the group as somebody's like, hey man, check out what I did. And they're happy and they're proud of what they've done and they show it. Hopefully that helps spark those ideas and now it becomes that more contagious spread. Any fears around that?
• 31:00
  Speaker B: Yeah. I think this is different from most IT solutions that we've all dealt with throughout our careers so far, where they're top down driven. In most cases where this is more grassroots effort. The end users are longing for this and wanting to use it and they're asking how do I do it? Right. And if you don't put the guardrails in place, that's the part that scares me a little bit. And hence why we're trying to put some of these guardrails in place ourself before we go out there and blow the horn too loud. Because when somebody gets excited about something, they're going to show their friends and if they get excited about it, but they're going to start doing things as well and being able to control that a little bit. Right. I hesitate a little bit because you want to control it in a way that secures your company, secures your data, secures your brand, that kind of thing. You don't want to cripple it either. So it's a little bit of a fine line right now. And I think that's a big challenge for us is training and rolling that out, encouraging use because there is benefits, but also not limiting it so much that they don't find the value.
• 32:03
  Speaker A: I'm trying to find a segue and I don't have an easy segue. So I'm just going to be blunt and switch a little bit. Part of what we're talking about is like just the training and enabling people and setting them up for that and then the data. The data governance is going to be key and we're hearing a lot about that. But I know that you raised a flag around another conversation. The processes. Let's talk about the processes and throwing AI at processes. What are your thoughts or advice or fears? What are your hopes and aspirations when it comes to processes and these tools?
• 32:41
  Speaker B: Yeah. So I think I was just at a conference in San Diego a couple weeks ago and I heard somebody there mention it in one of the sessions I was in. But it resonated well with me that if you grab an AI tool and throw it into your current process, you're likely just doing a bad process faster. And really those processes that we've got in the business world today are designed for a world that didn't have AI.
• 33:05
  Speaker A: Right. They were created and probably created before two iterations of technology anyway, even though it's not as game changing as AI is. But those processes and people still defend it with the. But this is the way we've always done it.
• 33:25
  Speaker B: Right? Yeah. And change is hard for everybody. Right. And especially when you start pairing that with, well, if you do that for me, what am I going to do? Right. What does my role look like after AI does this thing and you change that process? So then you're just kind of more apt to get pushback and fear. I think really, we all need to take a step back and look at our processes and as a whole and say, knowing what we know now, knowing the tools that we have right now, would we, or could we, or should we redesign the process from the start as a whole with AI being the center point and insert ourselves into the process where it makes sense versus other way around?
• 34:04
  Speaker A: Yeah. I'm thinking maybe this is a perfect place to start talking to that thought partner. Yeah. Hey, here's my process. If we were to redo this, talk to me about how could we do this better? Leveraging tools like yourself.
• 34:21
  Speaker B: Yeah, that's a great way to have that conversation with the thought partner of. Even if you have a process diagram, a workflow diagram, whatever, you can upload a PDF of it and say, sure of it. Yeah. Right. I need you to be a thought partner. Ask me one question at a time until you have enough information. I'm looking to redesign this process with AI as the forefront of the process or whatever, or how do we make this more efficient using the tools that we've got today?
• 34:48
  Speaker A: All right, I'm curious because one of the things that I've been doing with my prompts is I will tell the prompts as I'm trying to design something, I'll say, okay, no changes, no writing code until I explicitly say so. But ask any clarifying questions, ask me or present thoughts on this. Can we do it better? And I'll throw those in there, but I'll get like four or five questions. What's causing you to say, ask me one question at a time?
• 35:19
  Speaker B: Yeah. And I think that's part of one of the game changer moments for me was that piece of it. Because when you get five questions at a time and you answer those, your next question does not have the context of the first one necessarily so if you ask one question and, and you answer it, it says, okay, now based on that, here's another question, here's a follow up question. So I think you can get similar results, but I feel it just kind of focuses in a little better with the one question at a time thing.
• 35:50
  Speaker A: Okay, well I'll have to ask and try that a couple of times and see what the results are because I'm getting like five questions and I end up producing like a full page or two pages of answers. And I'll make sure to grab question number one and its answers and throw some context in and then question number two that it's prompting me with. And, and each of those questions have been distinct pieces of the whole of the goal. But I just found it interesting that you kept saying ask me one question at a time. So.
• 36:25
  Speaker B: Yeah, yeah. And the book that I mentioned obviously is where I got that idea from. I mean they explain it pretty well in there. But I think it's your question number one. You answer it now question number two has that context. So it might say, okay, based on what you said on number one, how would you answer this now? So it starts to build versus giving you five questions at a time. They're all based on that first individual prompt.
• 36:52
  Speaker A: One of the reasons that I've learned to tell it, don't take any actions, don't do anything until I explicitly tell you is because I've also had these tools, especially the ones that have more of that agency go off and just radically change portions of the code that have nothing to do with the current prompts and what I'm talking about dealing with or trying to fix at the moment. And so I've started to learn to tell it okay, you are only to affect code that has to do with this portion of the conversation, even though I'm leaving all of that prior context in that conversation. So. Because like I'm working on interactive summary webpage of something. So.
• 37:40
  Speaker B: Yeah.
• 37:41
  Speaker A: Have you cutting across anything like that or.
• 37:43
  Speaker B: Yeah, I think what I would do in that situation is explicitly tell it. You're not the doer, you're the thinker. Right. Help me think through the solution. Don't do it for me unless I tell you which, which I think is similar to what you're doing right now. Basically. Let's talk about it first. Let's generate ideas before you actually go do anything.
• 38:02
  Speaker A: Yeah, let's get to that solid plan defined, not just you run off and do things in between each one of our prompts.
• 38:11
  Speaker B: Yes, exactly. And I think that example is a perfect representation of why this is a hard con concept for people to grasp right now with AI, because depending on how you tell it to function, you might have really bad experience with it. So if you have a developer who's never used AI and they ask it a question, it changes all their code. You think they're going to use AI tools anymore?
• 38:34
  Speaker A: Yeah. And especially if they haven't backed it up or checked it into the repo or if it radically changes it and they don't have that beginning copy.
• 38:44
  Speaker B: Correct. Yeah. So teaching that context and teaching people how to interact is going to be extremely critical.
• 38:50
  Speaker A: I knew some of those things and I didn't even think about that when trying to build my first little agent to help me. It was the pain of having it radically rewrite things and destroy itself or destroy the agent that I'd spent lots of time trying to get to. And so I was happy with where I was at. Ask one little thing and then suddenly. Yep.
• 39:12
  Speaker B: And I would bench. Your first couple of prompts were one or two sentences max as you dove in right away. Till you have figure out for sure.
• 39:21
  Speaker A: Until I differed.
• 39:23
  Speaker B: Yeah. You dive into something and say, help me analyze my finances. Well, that can mean a whole lot of things. But if you're looking for. Help me build a personal budget based on this amount of income and this amount of time period and you get specific with it, you're gonna get vastly different results.
• 39:41
  Speaker A: Yeah, interesting. Okay, so kind of back to that use case that I threw out there of the cfo, CEO and CIO single transcript. And then learning how to get the different summaries and then maybe even feed those summaries back in and say, okay, summarize the summaries and give me a holistic. But yeah, the page long prompt or the page of context with the prompt to set up the overarching goal. Development and code writing. You guys doing any of that yet? Are you talking about that? Are you figuring out how you're going to do.
• 40:16
  Speaker B: We're doing a very minimal. We don't have a custom dev team. We try to. Maybe this is my own leadership philosophy, but I'm not a huge custom dev team guy. I think you can kind of handcuff yourself a little bit at times, but there's things like some power BI Dax queries and whatnot that we're using code for some basic spreadsheet formulas. We do a lot of design work with AutoCAD and there's Lisp routines and that sort of thing that are kind of per user in some cases. So we're using it for those types of things, but we don't do really any kind of what I would consider custom dev with your typical coding languages.
• 40:52
  Speaker A: So, yeah, all kinds of history and opportunities for me in the organization that I'm working with around all of that and trying to navigate those waters. However, this is about you and, and your experiences, so we'll focus in on that. So, okay, the training, get rolling that out. Are you rolling it out to all 600 in Mass? Are you going to run classes? Once you run classes, what are you going to do after that?
• 41:18
  Speaker B: So I think what we're going to end up likely doing is we've already got copilot free version kind of out there for all users to use. We've not trained on it yet. We will then. We were scrambling to make sure that it was authenticated into our environment, but everybody will have access to that. It's really the licensed version that we need to be strategic about. And when people are coming to us saying, hey, I need to use AI tools, my first question is, what do you want to accomplish with it? Right. Starting to ask those questions of what are you trying to do? Well, I want it to help me clean up my emails. Okay, cool. You can do that with the free version. Right. It's like, where does that line get crossed where we actually have to purchase a copilot premium or deploy kind of pay as you go agents for specific tasks. So that, that's kind of. We're going to have to ask a lot of questions throughout to figure out where it makes sense to buy a license version versus using something that's free.
• 42:15
  Speaker A: Yeah. Have you come up with methodologies for measuring the utilization of those purchased versions?
• 42:22
  Speaker B: We have not yet. Purview again, is going to help with that because we can get a lot of categorization on how people are using it, who's using it, how often people are using it. That's going to help us kind of govern the process and make sure that the people that we're paying for licensing for are actually using it and understanding what types of categories of things are they using it for. Again, I don't want to pay for a license version for them to create their home meal plan for the week. But if they're going to use it for something valuable, that's what it's there for.
• 42:55
  Speaker A: Yeah. And so when I first started doing it, I was asking the infrastructure team to keep an eye on that utilization and okay, we gave it to the executives. Which of the Executives are actually touching it and using it and, and who's doing what. They struggled a little with it because of course it's coming back and you can see who's touching it, using it just through the regular infrastructure, but it's coming back with a uid. And my team was not yet really used to transferring UIDs into usernames or government names and so they were struggling with that transformation. But once they recognized, oh wait, I can ask it to help me do that.
• 43:36
  Speaker B: Yeah, I think for us again, the adoption is really gonna come down to training. I've rolled it out to the 39 or whatever people that have it right now and some of those people aren't using it and a lot of times they asked me for it, I gave it to them and now I'm seeing that there's no usage there or very minimal. So then what do I do from a leadership standpoint? Do I just strip the license or do I go talk to them and say, hey, well you're not getting the value out of this. Why can I help with. And, and a lot of times you're going to get those responsive like, well, I, I tried it this one time, I didn't get a great answer so I just stopped using it or forgot I had it or I'm not sure how to use it. And then helping them work through that and that's where training is going to come in and we are going to do kind of global or I'll say company wide training. I've got a couple mechanisms within our organization that I do like monthly tech talks. I've got some internal channels that I can post to so some helpful tips and tricks. We'll create some good prompt examples that will get out there for people to say, hey, here, try this prompt. And I'm hoping that that kind of opens up some of the eyes on how better to use it.
• 44:43
  Speaker A: Yeah, and actually I, I was thinking myself of doing something like lunch and learns and I don't think center of excellence is a good one for this, but somebody else gave me one center of performance or center of utilization. Trying to just get the group of people who are using it in there talking to each other and being able to help each other so that now it's so new and our chances to use it in so many different ways are so large that you're going to be amazed at what the guy over in accounting is going to come up with. I know I need to be able to hear what he's doing with it because he's Going to give me ideas.
• 45:29
  Speaker B: Yeah, there's so much of that going on and we do have an AI user group that I put together and it's primarily made up of those people that have the premium licensing just to get them in a room like you said. And like what are you guys finding super valuable or what road bumps are you running into and share that knowledge.
• 45:47
  Speaker A: I wonder how many of the 39 that got those licenses are utilizing other external tools and how do we as the IT leaders figure that out without slapping on the mask that keeps them from. I'm thinking of the Silence of the Lambs, that mask you were talking about the handcuffs and I'm trying to think of moving it to that next level.
• 46:10
  Speaker B: Yeah.
• 46:11
  Speaker A: Cody, this has been a wonderful conversation. We have a question that we like to ask on. On you've been heard and that is what are we talking about or what are we not talking about today that you would make a prediction that in 18 months we'll be talking about this topic. What thoughts do you have in that realm?
• 46:32
  Speaker B: I think for me, one that I'm kind of watching closely is the pricing model around AI and AI usage. AI is changing so quickly right now and there's so many startups and so many companies that are adding on AI tools to everything that nobody really knows how to price it yet. And the way that they're pricing it and selling it today is not cost effective in most cases from what I'm finding. I'll use Copilot as an example. Right. The Premium version is 30 bucks a month per user, $360 a year. Yep. I can make a pretty good use case that I get that much ROI pretty quickly. But if I take that times 650 employees, that's a big number real quick. And that's only one tool. So now if you like, well, ChatGPT does this thing better. This pocket of users also needs to have access to ChatGPT. That's another 30 bucks a user per month. Right. Like this thing can get out of hand in a hurry. And now we're seeing the usage being paid in tokens, which I joke that this is monopoly money at this point because nobody knows what that means. You can't measure that. You don't know how much a prompt is going to cost you in terms of tokens. So I think the pricing side of this is just a kind of a disaster right now. And I think there is going to be a bubble, if you will, where it's going to drastically change in the next 12 to 18 months, I think. And it has to, as people start to roll this out there, people are going to realize that it's, it can get really expensive real quick. I mean, again, Copilot, if I was to roll it out to all of my users, our Microsoft licensing is going to be 50% more than what it is right now, just overnight with one tool.
• 48:21
  Speaker A: Right. And, and actually I did a little bit of a calculation off of that. If at that $30 per month, as long as there's no token cost behind Copilot, if somebody that's making 60,000 a year, $30 an hour, if you can save them $15 a month, then it pays for itself. And it was surprising to me. But that $15 or that 15 minutes per month or, or was it per week that they constantly got to keep trying to save themselves that amount of time?
• 48:55
  Speaker B: And how do you prove that we've done that? Right. How do you prove with data that you've actually saved that time and that they're not taking off 15 minutes earlier or whatever the case is?
• 49:05
  Speaker A: Right. So I can do little bits here and there. Hey, I was able to create this presentation in an hour instead of the seven hours that I was foreseeing it taking.
• 49:14
  Speaker B: Yep.
• 49:14
  Speaker A: But once I get to that level of productivity, now I've got to ramp it up from that level of productivity. I've got to keep ramping it up to continue to pay for it.
• 49:24
  Speaker B: Yeah. And I think when you throw the token thing into the mix, I think Microsoft's, at least the last I looked, they sell like the per use tokens.
• 49:31
  Speaker A: Right.
• 49:31
  Speaker B: You can buy a pack of 25,000 tokens for $250. And the pay per use model, I think they say, is a minimum of 3 tokens to 12 tokens per interaction.
• 49:44
  Speaker A: Cool.
• 49:44
  Speaker B: I have no idea what that means. But next year when they change their pricing, they're not going to up it from $250 to $300. They're going to say your minimum is now 5 tokens to 20 tokens. How do I budget for that? How do I know what that's actually going to do to my bottom line and how do I plan for that? You just don't know.
• 50:04
  Speaker A: Yeah. Because you're going to have that huge disparity in the users. The person that's going to put in the 8 to 10 word and the other person that's going to use 5 to 6 attachments for the context and then ask that 10 word question. Radically different cases. Radically different token usage between those two cases. Yep.
• 50:27
  Speaker B: And, and now we've got an AI tool that we're putting on our financial system right now that uses tokens as well. But their token model is not the same as Microsoft. So we get I think 50,000 tokens per user per month for this pricing model. So now I'm looking at overages and it's going, well, it's $3.50 per million tokens over like well, $3.50 isn't much. But when I use it in a matter of three prompts, I'm doing math, going like this thing is going to get out of hand, right. It's going to get ridiculous real quick.
• 51:02
  Speaker A: So it's going to be exponential.
• 51:04
  Speaker B: That's why I say it's kind of monopoly money right now because nobody knows what the actual currency is. I think there's more to come on that in the next 12 to 18 months I think that's going to look radically different because people are going to have to get competitive in order to get these tools into place. Nobody's willing to sign on to long term agreements or big price tags right now when things change daily.
• 51:27
  Speaker A: So yeah, and they're changing daily. They have new models, new things. So as we come to a close on this, what is the one thing you want to make sure the business and, or your peers or the people following us into the leadership of IT in the you've been heard community, what's the one thing that you want them to hear from you?
• 51:53
  Speaker B: Yeah, I think I'll circle this back to the start of the conversation in kind of the listening. And I, I go back to a quote that people don't know how much you don't care how much until they know how much you care. It's a quote that I heard, I think John Maxwell is who said it, that I heard it from. I don't know if he got it from somebody else. But listen and understand that and, and allow people to understand that you're here to help make their lives better. And you want to use technology as a tool to do that, whether it's AI or anything else. Being that strong business partner and letting people understand that you're there to help them is going to be critical, especially in this AI world right now where nobody knows what to believe, nobody knows what it's possible and what's capable of. And I think that's really going to go kind of a long way when it comes to this. So we don't need to walk into the room and speak technical jargon and try to prove that we're the smartest person because we know big words. It's more of how do you bridge that gap and help the business user understand what AI even is.
• 52:54
  Speaker A: Yeah, I love that closing thought. So thank you very much, Cody. We truly appreciate you sharing your time, your experience and what you bring. Awesome.
• 53:04
  Speaker B: I appreciate you having me on. It's been a lot of fun.
• 53:07
  Speaker A: Thank you for taking the time to be on. You've been heard.