Latest / You've Been Heard / 458- Stop Turning the Light Off w/John Errington

Transcript

458- Stop Turning the Light Off w/John Errington · You've Been Heard

• 0:00
  Speaker A: The analogy I've often used for teams that I manage and they hate to hear it from me, is okay, the light goes off and we hit the button to turn the light off. Like that's what we think we're trained to do. That's what our value add is. So it's like, hey boss, I'm turning this light off every day and it's great. And I'm like, guys, that's not the value add. That is the thing we should be automating. That is the thing we should be taking the human element out of. What I need you guys doing is asking why the light is going off. Get to the root cause of this issue and, and make that what you're putting your time and effort into fixing. Not just turning the light off. Turning the light off is not the value add. The value add is fixing the core of the problem.
• 0:46
  Speaker B: After talking with IT leaders, I've come to discover one thing. And it's the same thing, you know, it's the same thing everyone else knows.
• 0:54
  Speaker C: What?
• 0:55
  Speaker B: Whether they want to admit it out loud or not, nothing gets done in any company without it touching it. Nothing. But here's the thing. When everything's on fire, when nothing's working, when the ransomware hits, when the whole business grinds to a halt, it is completely indispensable. You're the most valuable asset in the company. But when everything's running like a well oiled Tesla, you're a line item, a cost center on a spreadsheet. You've done the work, you've built the career, you've earned your seat at the executive roundtable. But having a seat at the table is one thing. How much nicer would it be to actually be heard? This is you've been heard.
• 1:40
  Speaker C: Welcome back to you've been heard. Sitting in the guest chair today is John Arrington, an executive leader who knows how to cut through organizational noise and build alignment. John, mind kicking off things by introducing yourself and sharing what your ride to the leadership chair has looked like?
• 2:00
  Speaker A: Sure. John Arrington. I've got over 16 years IT operations leadership experience over shoot, 27 years IT experience in general. I kind of fell into it accidentally. I started working for pharmaceutical a couple years after I finished high school. I didn't go to school right away. Started off in their records management department and I worked with some older ladies in that department. And since I was a young man that knew how to turn on a computer, that made me the de facto local IT guy. So I was part of a few application conversions where we were going, this is in early 2000s we were going from mainframe applications to Windows based applications for some of the stuff we were doing. From there I oh yeah, it was a great time. And then I took over the electronic repository of our clinical study reports and managing that, I did that for a couple years. I slid over into hr, did a few more mainframe application conversions in that space and then from there I moved into our data center and it proper and was doing the standard server stacking and racking tape monkey, changing out data backups and dealing with service calls and all those things. During my time there unfortunately my role got outsourced as the pharmaceutical got purchased by Pfizer. So then I moved over into the E commerce space over there I joined the third largest e commerce retailer at the time in 2010 which was GSI Commerce. As I joined them they were a Linux base shop and I had very little Linux experience coming in. I made that clear. But I was thankfully working with some really smart guys that were willing to share their knowledge with me. So sitting down and looking at these guys, looking like movie hackers at the command line, writing all this stuff that I had no idea what they were doing. I just shut my mouth and kept my ears open. So I quickly saw a value in taking over those processes, making sure that I was documenting everything, making sure that I was helping to create an onboarding plan for the next guy that came out behind me and make sure he was able to hit the ground running and get speed in short order and doing all that. We had new leadership come into that organization and evaluating everything. He tagged me as one of the leaders so he drugged me into IT leadership kicking and screaming because at the time I did want to be one of those individual contributors on I'm like, I don't want to be responsible for other people's stuff. I'm going to do my stuff and that's it. He made it very clear you're already taking care of other people's stuff and you're already doing the job. I'm just basically giving you the title that goes along with it.
• 4:37
  Speaker C: This was 20 what? Because it was 2010 when you started with them.
• 4:46
  Speaker B: If your Internet goes down and you're calling carriers yourself, that's not strategic it we help teams centralize monitoring, proactive tickets and escalation across every location without capex. Less noise, more uptime, make it boring again. Just go to you've been heard.com and answer the seven questions to simplify, streamline and save.
• 5:07
  Speaker A: That was about a three year window. So it was around 2013, 2014 or so that all this was going down. And what he also did. So he put me in a position of leadership, but what I had to battle myself with was that individual contributor mindset. And at the time, that operations team, we managed everything through an email inbox and it was not good. So at the time we would spend more time making sure that we filed an email in the correct folder so that the right team got it and we were doing actual work. And so he called that out as a risk right away. And he's like, you guys got to get out of this email inbox, get your alerts into monitoring and into your ITSM and get out of the email space. And we're like, we're too busy, we don't have time to do that. The business is going to be impacted if we take time out to do that. And he let us drown for a month or two and then Buddy would keep staying on us about it. And finally I took one of the alerts, I went to the Windows team and I was like, hey, we keep getting a space alert. We have to keep clearing out logs. Can you guys help us out with this? And put a script in automatically, clear out those logs and you'll be in good shape. And it was like the light bulb went off for me. And I was like, he was right, we can make things work better for us. And then I just went on a terror like attacked everything. And it was like I was addicted to fixing things at that point.
• 6:25
  Speaker C: You went from the Linux world where everybody was doing everything, command line, started filing the emails, went over to the Windows guys and they're like, yeah, just write a script for that.
• 6:35
  Speaker A: Well, and that's where like the Linux guys are like, I'll just fix it and make it go away. I'll just fix it and make it go away. And the analogy I've often used for teams that I manage and they hate to hear it from me, is okay, the light goes off and we hit the button to turn the light off. Like that's what we think we're trained to do. That's what our value add is. So it's like, hey boss, I'm turning this light off every day and it's great. And I'm like, guys, that's not the value add. That is the thing we should be automating. That is the thing we should be taking the human element out of. What I need you guys doing is asking why the light is going off. Get to the root cause of this issue and make that what you're putting your time and effort into fixing. Not just turning the light off. Turning the light off is not the value add. The value add is fixing the core of the problem. So I ended up taking over that team managing it. We got purchased by ebay Enterprise. Along the way, ebay tried to dip their toe in the e commerce market. Realized it wasn't for them and then they sold us off to a Belgian company. A Belgian company wanted to then kind of strip us down to our base elements. So I took that operations team and rebuilt it with contractors a few times. Had some pretty good success. But every time I'd get the team rebuilt, Comcast would come along and steal all the engineers that I trained up and take them on their Viper team. And I basically built half of the Viper team for Comcast with the contractors that I trained. So after a couple years of doing that, I decided to move on and moved over to qvc. QVC at the time had just purchased Home Shopping Network. Home Shopping Network also had another company that they were owners of as well. And so I was brought in to consolidate and modernize their IT operations, which was not one IT operations, it was three IT operations wearing a trench coat. And they all expected their own white glove service. So a lot of things to take apart. Standardize, process, improve. Did some great job there.
• 8:27
  Speaker C: Sure they loved you for that once you were done, but all of the teams for you to standardize and make everything simpler and easier for them one
• 8:36
  Speaker A: day and then I was the villain the next day depending on who I was interacting with. So did a lot of great work there, both internally with the team, with career development and moving to folks around and externally with how we supported the organization. Really modernized that operations team. And then along the way they also handed over the workload automation, job scheduling and file transfer. Teams basically do a lot of the same things. They were stuck in a situation where essentially the inmates were running the asylum and the developers that we supported dictated how we did the work there. So we really came in and provided some guide rails and some self service and really turned that process on its ears and, and made a lot of good progress. And then I took over the change management process with the intention of doing the same things during that. We then unfortunately had some of our staff from an L1 perspective outsourced to Accenture. But that actually ended up being a blessing in disguise because there were some initiatives that I was trying to drive that I just wasn't getting the buy in from the resources engaged. And when I took those resources out and put essential resources in there and laid the plan out, we made a lot of really good progress. I'll be honest, my previous experiences with Accenture were not what I would call positive. So I went into it maybe a little bit pessimistic, but I didn't show it. But I was pleasantly surprised at the results that we got there and left that place in a lot better condition than it was when I joined them. And then this is now my starting my fourth week at new position here at the Handling systems. And I'm very excited about the work I have in front of me here. So I get to leverage a lot of the experience and lessons I've learned in the past, coupled with a new industry and new challenges to keep it interesting.
• 10:29
  Speaker C: So, man, I got like a good 10 or 15 different things to hit you up on and ask you about, but one of the things that you just said really kind of piques my interest the most, and I want to hit that first. So as you jump from industry to industry and you're already trusted in a leadership role, what's your playbook to learn the industry because it's so much at the leadership level, within it, it's so much about understanding the business. And so you're moving into a completely new industry. Obviously you know about debits and credits and making money and those portions of it, but you're walking into a new industry. What's your playbook like right now? What are your primary thoughts at the moment as you try to figure out where the quicksand is and where the solid places to stand are?
• 11:28
  Speaker A: So for me, the needs of the business have kind of just always been easy for me to kind of innately understand. I always look at the business as if I was running this business, how would I run it for myself? And it's obviously you're looking for cost efficiencies, process improvements, and spending money like it's your own. So understanding the needs of the business has kind of been the easy part. Understanding how the business works and what drives the business, that's where your challenge comes in. And for me, it starts with the people with their boots on the ground. So my previous role, I'm in operations. So I started with my operators and would sit down with them and do a ride along and do a day in the life and see the challenges that they were facing on a day to day basis. And then I would map those challenges back to whoever owned the process that interacted with my operators and then Build the relationships between those two teams. And my experience in operations, we've often been looked at almost in an antagonistic manner because as operations, it's often our job to kind of call out when something's not working correctly. That same boss that I had that drugged me into leadership kicking and screaming. One of his skills that he said that I have was the ability to tell people that their baby's ugly. And operations were jack of all trades, master of none. So we need to know a little bit about everything across the infrastructure so that we can support it. And then when we have a problem, we escalate to the SMEs. But those SMEs, they often get tunnel vision. The engineers want to do engineering work, the network guys want to focus on network DBAs. They're in their space and that's all they're worried about. And when you're coming to them and telling them, hey, your database is having problems, you're telling them their baby's ugly. Put a lot of work into that thing. And some operations guy that doesn't know anything is coming and telling them that it doesn't work properly. So having the ability to do that, which I've always just leverage. Listen, I'm showing you the data that I see here. This is what the data says is happening. I may not have the whole picture, I don't have all the answers here. But you have to explain why the data is showing me this information, because this is what I'm seeing is impacting the business. So then going back to your question for now, this new industry that I'm not experienced in, because prior to this role, I'd spent 15 years in E commerce. So I mean it was just Miami. I knew the business top the bottom. So the transition from GSI Commerce to QVC was very smooth because it was already an industry that was in. So now I'm in a business in which we're dealing with material handling. So we're dealing with forklifts and equipment and shelving and conveyors and automation. So I need to know what the guys that are working with those things, what their challenges are. So far in my three weeks here, I've done some ride alongs with some of the field service techs. I go down and I see what doing the guys working on the forklifts, I see how they're getting their invoices and how they're working on a phone and entering the information for their invoices and seeing all the challenges that they have at their level and then coming Back and working with my team to say, hey, this is what I'm seeing. These text challenge, these texts being challenged with what can we do as an IT organization to make their job easier? Because at the end of the day, they keep the cash filter store open. They are the ones making this business money. It's our job to make their jobs as easy as possible so they can turn our service calls around as quickly as possible so they can get to the next one. So we have happy customers, we have satisfied techs that are able to efficiently and easily do their jobs. And then probably all of that with both the techs and my own organization. I think the most important thing is making those individuals understand how their role plays a part in a larger role of the business, how they're part of a larger overall ecosystem and how what they do at their level on a day to day basis, how that pays off at the end of the day on the balance sheets that get reported out to the financial industries.
• 15:37
  Speaker C: That's a great approach. I mean, there are so many different things, so many different challenges and different opportunities to help out all of those different teams as long as you can see them and as long as you can help deliver on those. So one of the other things that you mentioned was you got a bunch of people, at least in your past, you were talking about all of the guys with the hammers going around hitting the nails, if you don't know the analogy. And I'm speaking more to our audience than I am to you. If you got a hammer, everything looks like a nail. So the network guys are focused on the network, the DBAs are focused on the DB, all of those. How do you help break them out of that? How do you expand their world? How do you get them to take the, what was it, the red pill or the blue pill?
• 16:26
  Speaker A: So I would do that two ways. One of the first things that I would do would be from a career development standpoint with the teams that I'm responsible for. So career development and personal development for my people is one of the most important things to me. Again, another element that I never thought that I would care about before I was a leader, but has really become the more satisfying part of my job. And so being in operations, my goal was to always considered operations to be the feeder mill from an IT perspective, from a staffing perspective. So ideally you have somebody come in from a help desk, service desk perspective, they learn high level how the organization works. To me, the next logical progression is your operations team. So you get some L1, maybe some L2 experience in there really start to get your technical skills sharpened up. And then if you see if you have an aptitude in one of those certain areas as a leader, I was always trying to help people to develop into those areas because if I could get somebody to become an engineer with an operations mindset, they're going in with that big picture mentality, understanding that they're part of a larger overall organization and they can influence the rest of the people within that team. Obviously you don't always get to develop people and put them into those teams. So the other thing that I would often leverage is just relationship building. So I made sure that I had regular one on ones and interactions with the leaders in those spaces. I made sure that when we had incidents that need to be resolved, that whether my team, the operations team, was engaged or not, we were following up and making sure that we were learning what was happening there. We would often say to these teams, all right, hey, we had to escalate this to you. Is there anything that we could have done from an operations perspective? Are there things that you can show us that we can not wait your engineers in the middle of the night and make sure that we're trying to do everything that we can to get the business up and running again without escalating and building that relationship with the leaders of those groups. To say, hey, work with my operations team, allow us to take that low level, easy stuff off your teams. You're taking the load off of them and they can focus on the after hour support when they get called and all those things. And then you're putting more tools in my team's toolbox and making them more valuable resources for the organization overall. And then that shows, hey, I'm not just here to worry about my stuff and that's it. We are here to make everything work better for everybody. So consensus building and relationship building is key to be able to get that kind of buy in for the organization.
• 18:58
  Speaker C: I've realized that I want to circle all the way back to the very beginning of the conversation. It's something that you said that I wanted to bring up and get a little bit of definition to you or you mentioned your experience that you have 15 years in IT or Ops 27 years in IT overall. Help define IT Ops or IT Operations to some of those that are listening that have only been doing networking or only been doing one of these blinders on specialties. And then tell me a little bit more the size of the organization where IT ops really kind of comes into play. Or what drives bringing that kind of a role into an organization.
• 19:46
  Speaker A: So IT Ops to me is what ties all of your operations together from an IT perspective, obviously. But I think it really is your thread that blends IT and the business. IT in particular, for decades has kind of been this thing off to the side that's always been able to operate on its own without much oversight for two reasons. One, it was necessary for the business to be able to conduct their business. And two, the business leaders, when they would start to try and interact with the IT leaders, they didn't have enough background and understanding of what the technology was doing to really challenge IT leaders. So it was kind of always this thing that operated in a bubble of its own. Like, particularly when I was at GSI Commerce, we dictated to the business how it was going to do its job. Like, we told them the equipment we needed, we told them whatever service that we needed, whatever outside vendor that we needed, we dictated it to them. And the business would just say, write the check. Okay, you guys need that? We'll do it. As we approached the 2020s, I started to see a shift in business and really start to incorporate IT as just another one of the pillars of business, just like finance, hr, all the. And really start to put that kind of oversight in place. I think that was the best thing to happen to IT operations because like I said, IT is what fills the space between IT and the business. So that's why, as an operations leader, that's why I stress making sure my organization, the people that work for me, understand their role and the larger overall business. And it's important that you make it so they understand what they're doing on a daily basis, how that impacts the bottom line for the business. So IT operations really ties all those different areas in IT together to then, you know, demonstrate how important IT as a whole is for the business.
• 21:46
  Speaker C: Okay, because to me, I was thinking of IT Ops being, I don't want to say subset, but I can't think of a better term of it, that grouping of technology where, like you talked about the forklift operator or the guys out in the warehouse or the guys out on the production line, and I'm thinking about the technology that they're using specifically that it's not, yes, everything's networked, but it's the forklift operator and the pick and pull and the workflow that goes into that. And then like on a manufacturing floor and all of the technology that's out there, all of the robots and all of that hardware and that kind of the fine tuning of those technologies and working with that kind of technology. That's what I was kind of thinking when I was thinking of IT Ops and now I hear what you're talking about. And I've had this discussion with a couple of leaders and people who are working for those organizations that go out and do research for it. And we've been talking about this thought of IT is going to kind of disappear as the way it is today. We are going to be out there with everybody, we are just going to be other members of the workforce and actually the whole workforce is going to be IT to that level. But you're still going to have your infrastructure type people who are running the cables and understand the networking because there's so many people who leverage their cell phone every day but have no clue what is behind it that makes everything work. But there's still going to be that level of it. But IT like you were talking about in the bubble is going to disappear to a certain level and we're just going to be sitting right next to each other out there learning how to place the order, how to optimize the
• 23:44
  Speaker A: workflow, whatever workflow, it happens to be a hundred percent. And it's like I said, we one, nothing, almost nothing happens with business at this point without it's involvement. I mean we are spider webbed into everything because it's just the nature of the world that we live in. Technology is everywhere around us. I mean it's how all business is done to your point. So that's where we as IT leaders, for those of us that get it, I think become that much more valuable. Because in IT you may experience this, I'm probably sure you have most guys in it, particularly the really skilled engineers and things like that, they get that tunnel vision like I mentioned and they just want to come in, they want to do their work, they want to do the technology work that they've done and they want to knock off their assignments and their projects as they get them without really much thought or concern about what the business is delivering and what that particular project is going to deliver for the business. They know that they got these requirements, they need to set up and do whatever work that they need to do and then they hand it off to somebody and then they want to just walk away and go on to the next thing. IT leaders are that much more valuable now because like I said, understanding that meld between the business and IT and then being able to get those individual contributors to understand that is critical now. And I'll also be honest, it does attract a certain subset of the population that is a little more prone to being on maybe the autism spectrum. And to have some approaches mentally to how they do their job, that limits their ability to kind of understand that role. Not so much limits it, but their inclination is to just do what they've always done and focus on the things that they focus on. And we as leaders, leaders in it, that's a difficult space for us to be in because they're adding value and they can become frustrated when you're trying to make them understand that, hey, I know you want to do it this way, but the business has decided we need to do it this way. That can be a very challenging place to be with some of your more skilled engineers because from a technology standpoint, their perspective is often, who are you to tell me, I know this better than you or the business? I should be the one telling you how we're going to leverage this technology or this approach to things. And most cases you're going to be right. However, the business also dictates that certain decisions have to be made and we have to follow that. And you may not always agree with it, but we got to deliver, right?
• 26:17
  Speaker C: And it's not about how the technology is leveraged to do the work. It's how the work is done and uses technology to advance the work. And we forget that. We lose sight of that so often.
• 26:33
  Speaker A: I think you hit on something important there is that that all it exists to deliver for the business. And there are often people that are in it that are so in the trenches and so down in the weeds, they're like, well, my value add is to deliver this thing. It's like, okay, yes, well, you are part of a larger organism. And the organism, the business, is what drives what you do, not the smaller component of it. It's right.
• 27:00
  Speaker C: And unfortunately, what was it? I think it was last week or the week before, I had to let somebody go because they only had those blinders on. They could not expand or would not. I think it's more of a would not. They could work with the system and they could do the things with the system, but when it came to the context of the organization and the understanding of the data that was going through the system, they had no desire to learn anything more. And so. Well, why would you do it that way? Well, because if you don't do it that way, then this happened and that happens and somebody gets hurt. And you gotta know how it fits.
• 27:45
  Speaker A: And you get so many of those people that Just like you explained. Like, they want to come in, they want to make their widget, and that's it. They don't care about what the widget is a part of. Like, I'm making my widget. That's what I went to school for, and that's it.
• 27:57
  Speaker C: Yeah, you definitely have to have the understanding of the context. I find myself still surprised a little bit at the difference in our definitions of operational it. And so I'm glad that I asked because I understand what you're talking about. And yes, it is critical and it's where we're headed. And it's not that subset that I was thinking of, just that type or that partition of technology of where the operations group is leveraging technology to accomplish the goal. It's the leveraging of technology. Well, I'm just going to repeat myself and undo what I just said, but it's use of technology to complete the goal across the whole organization, not just within operations.
• 28:46
  Speaker A: Yeah, it's like application versus philosophy.
• 28:50
  Speaker C: Yeah. When you said application, I'm thinking, okay, I've got the application group, I've got the infrastructure group applying. But yes, so that does kill one of the other questions, because I was thinking of that operational piece of IT and thinking that, okay, how large does an organization have to be before they start to say, okay, we're going to develop or we're going to spin up a group of people that is just going to focus on how like the warehouse technology and the forklift driver technology, and in my case, the truck driver technologies and those pieces of it. Because I know organizations get that large, that they do need a group just for that. And you and I are still in that arena of we deal with everything for sure.
• 29:36
  Speaker A: And it's depending on the industry and how the business grows. A lot of times your IT departments, they grow up as these individual tiers independent of each other. Like, they only connect and they only talk to each other when they absolutely have to. And that's how you end up with the silos and all this people just not working well together. And then there are often companies that don't even have an IT operations team because it's not really a concept to them. They have that mindset of, oh, well, I know I need an engineer, I know I need a dba, I know it ain't a networking guy, and. And turn them loose thinking that they can all handle and maintain their individual components and IT will just naturally work together. If you don't have that oversight over all of it, it quickly becomes the Wild west. And it will quickly impact your business.
• 30:22
  Speaker C: Yeah, because then they start having fiefdoms and fights over who owns what and when things break, who's going to fix what and who was at fault. Versus, okay, how do we get past this? What do we do to minimize this?
• 30:39
  Speaker A: What happens is that environment then breeds the not it mentality, whereas you're having an outage going on. If you call the networking guy onto the call, he'll go, yeah, everything looks good from networking perspective. See you later and drop off. And then, all right, now I gotta chase the engineering team, I gotta chase the database team. Whereas I developed a method and a mindset and my operations teams where if we were having outages, major incidents, we took the SWAT team approach, everybody jumps in, everybody's looking at what they're responsible for. They're reporting in on the call that we're on. And then they're saying, all right, I don't see anything in my space. What do you want me to look at? How can I help? And then they would stay and remain engaged until we cleared them. And we were able to get commitment to that approach because we proved that it had been successful in the past. We'd had so many business impacting outages that were elongated because we're trying to. Johnny's on vacation and we paged him three times and he's not answering and it's not going to his backup. And yada, yada, yada. We took that all hands on deck approach with a SWAT team approach and say, everybody comes in, reports on their functional area and then we all stay until we get it resolved or we fully vet out that no, it's not networking related to network engineer. You can go. That took an industry shift because when we first started doing that, you did not want to hear the engineers when they got on the call saying, I'm 100% certain this isn't me. I don't know why we're doing this. And then all it took was one time where that engineer would get on the call, he's like, oh, wait, all right, we do have a networking problem. Now I see why you guys are calling everybody. And just like me with the email thing, all it takes is getting that light bulb sometimes to go off for folks to get the buy in. And then you hope they become your advocates. And then they're infecting that the people that they're working with every day. And then the organization starts to get it and the fight isn't so hard
• 32:35
  Speaker C: anymore, so we can't escape this conversation, you probably knew it was coming. How are you leveraging AI or how are you blending AI into your IT ops today? Or how do you plan on blending it in tomorrow?
• 32:53
  Speaker A: So I look at AI just like I looked at automation when it really started to take off. It is a tool and you're the tool needs to be leveraged and it needs human oversight. Just like automation is not the answer to everything and didn't make everybody's job go away, AI is not the answer to everything and it's not going to make everybody's job go away. Is it going to make some for sure. But I think it needs to be leveraged in the proper space and it needs oversight and it needs a human at the decision making part. Depending on the area, depending on what it is, obviously you can fully automate some end to end processes, but your critical ones, the things that could have an impact on your business, the things that could close the cash register, you still need that human element and that oversight of it. So where I've leveraged AI so far in the previous role, so we had a major incident process and every week on Monday we would report out on a previous week's major incident. So we capture all the incident impact, what the dollar impact was and then what the impact to the business overall was. Gathering that data and formatting it and then packaging it up in an executive palatable way. That could sometimes, depending on the complexity of the incident, that could be a multi day effort for the problem managers that ahead. So all of those calls that we would have for those major incidents, we would have recordings of. Once we put AI in place, we would essentially have it do a postmortem of that recording and kind of give us an idea of what it thought the issues were, action items taken, roll call, all that stuff. And it really took a lot of the data elements that my team was taking three to four days to kind of comprise and take it down to a 20 minute process. Now again, like I said, you still need that human element. I still needed my problem managers to review all that for accuracy. How often AI has hallucinations and mishears things. And so it took a long process, shorten it down and then it allowed them to just really kind of focus on the oversight of it, cleaning it all up and then we standardize that. So now they're going to be taking that and moving to the next level, whereas we have major incidents, it's reviewing all that stuff. It'll now be looking for, has there been a historical occurrence of this could it suggest solutions? Could it put things in place to be able to resolve that? And then what we're doing, what they were doing over there with that next is we're identifying what suggested solutions actually were successful. And then we'll put automation in place to kind of execute those given if it meets all the criteria of the scenario and we can put in an automated solution to restart a service or whatever it may be, we'll take those lessons learned and then again put AI in place to be able to kind of handle not only capturing what's happening, but also trying to take the steps to solution as well. So I think it has to be iterative like that. I think you have to be very careful and deliberate with it. But I think AI has its strengths. Data correlation, data cleanup, hygiene, all that kind of stuff. Standardization, de duplication. I mean, you cannot match the performance it has with a person. It's just not with a team of people, could you? But you still need the oversight.
• 36:20
  Speaker C: So with the oversight, with the leveraging, are you trying to control it or are you planning on trying to have your arms wrapped around that beast from the beginning? Or, or how are you keeping the CEO and the C suite from just jumping in and having their own subscriptions and just doing whatever they want with Claude and wherever, and then coming to you and saying, make this happen.
• 36:46
  Speaker A: You're not controlling that. And there's the particularly in it, you have leadership that at least has some technical skills. So they can be very dangerous in that space. That's where operating with the mindset, an approach of having guardrails on everything that you do, having oversight on everything you do, worked in very regulated and audited environments. Security obviously is number one on the list when it comes to anything new in the industry. So with all of that in place, and prior to my role here and being in operations, we were the custodians of production, we were the gatekeepers of what gets into production. So checking off all those checkboxes, the security aspect, the oversight, the control, they're just innately built into how I approach everything. So it's like when it came to AI, it was the same thought process as always. It's like security first, then controls and guide rails and then self service as much as possible. If you can leverage these tools to be able to have your organization safely implement those, by vetting out who you're working with from an AI perspective and then putting in those guide rails to decide how much they can and can't do, you, you protect the people from themselves. So to speak. So you, you really have to think of. And it's tough, particularly with emerging technology like AI, to think of everything. Like I think back to when APIs first became in vogue and our developers were going crazy with how easily they could connect everything and they wanted to like cram all these things into production. And I'd be on the CAD meeting and I'm like, guys, I don't see anything security related in here. Like, do you realize what kind of attack vector you're creating here with something being able to ride an API in here? And you guys didn't even think of security. Like, security's always gotta be in everything that we do. It's not just one team that's responsible for that.
• 38:41
  Speaker C: Yeah. It's not just the networking team that can punch holes through the firewall. Okay. So one of the things that I'm starting to recognize, and I think many, many of us are, is that some of the governance, I keep hearing those words, governance, and I'm hearing like AI governance, but I'm also hearing data governance in ways and as a need that I haven't heard before. Governance is kind of like that high availability and disaster recovery. It's that thing that, oh yeah, we want that. Just don't make me pay for it, but it better be there that day. I need it.
• 39:19
  Speaker A: And don't hinder my business either. It can't slow us down at all. I want the protection, but I don't want any of the slowing down delivery.
• 39:27
  Speaker C: Yeah. And so the governance, getting everybody in the organization involved in the governance to identify the value of that data, of how critical it is and where it belongs and who owns it. Because there's the other portion of data governance that we never used to talk about was, okay, this data is operational data and it's bad data because somebody wasn't paying attention as they put in times or they didn't care and nobody slapped their hands for putting in delivery times one minute after a pickup time. So that governance is becoming super important nowadays with AI and now the organization is willing to invest some time in it.
• 40:12
  Speaker A: Well, and I think because the problem has been compounded in that prior to AI, that data for the most part was still under your control. For the most part, whether you're in the cloud, you're on premise, like wherever that data exists, you kind of own how it's leveraged and how it's managed with AI now, and putting that data out there, you're putting that data in another company's cloud for lack of a Better term, it is going out. And they're. Not only are they using it to deliver for you, but you're crazy if you don't think that they're using that data to continue to train their AI, to continue to make it work better, to leverage whatever you're leveraging AI for, to see if they can apply it somewhere else. So there is a lack of control over the data that I don't think a lot of organizations are thinking of, particularly from a leadership perspective. They're just thinking productivity. And I can get this, and I can get this. But again, like I said, the security aspect of things kind of falls by the wayside, particularly with new technology, because they're so focused on ability and what it can do for us. We're not thinking about protecting ourselves.
• 41:22
  Speaker C: Yeah, well, circle back a little bit to what I was talking about of the CEO or one of those guys with the corporate card who spits up their own account, starts doing their own stuff, they're vibe coding some kind of an application comes in, shows it to their buddy that's sitting in the cubicle next to them or the corner office across from them, then they start using it, and next thing you know, it's all over the place. And yet when you right click on that web page that they're using for this stuff, oh, here's an authority token that allows me to act as if I'm you and have full access to that system because they're not thinking about that security level just like you're talking about. What are you doing to deal with that challenge? So you're only four weeks in. So.
• 42:12
  Speaker A: So here I haven't really got my hands around it just yet, but in the previous organizations, I think that's where your change control process needs to be really mature and it needs to have engagement and oversight by the entirety of your organization. A lot of change control boards, in order to kind of expedite things, they try and keep the board a little bit smaller because the more people you add, the more talking and the slower everything goes. But it's critical to protect the business, and you have to take that time and you have to have the right people involved to protect the business. And not everybody knows everything, not everybody's an authority on the entirety of the organization. So that's where it's like. You need representatives from the business, you need representatives from technology. You need everybody to have their chance to have a say. And what this thing is going to do is you're getting ready to go to production and having the Controls in place where nothing gets into production without that oversight is critical. And, and I've been in shops where we've done a good job of that and I've been in shops where we haven't done too good a job of that. So it's a constant challenge.
• 43:21
  Speaker C: Oh yeah, for sure. Yeah. I've been in shops where we start testing something and we do it in the production environment because we need some of that data. Next thing you know, that test is now production and you can't stop doing it because they saw the value, they grabbed it and they ran with it.
• 43:38
  Speaker A: So I've been in the position plenty of times where it's been, what is this alert and what is this service that we've never heard of? And you start digging into it and it's like, oh, that's a new thing we just put in last week. And it's like that would have been nice to know about and that would have been nice to be able to support.
• 43:55
  Speaker C: We like to ask our guests to make a prediction. We'd like to hear what you think that we as IT leaders will be talking about in 18 months that we're not talking about today.
• 44:10
  Speaker A: It's obviously going to be AI related, but there's still so much uncertainty from an AI perspective on who's going to be the big dog. And the other side of that is like you hear these stories about rogue AI hacking other companies and all that, which I put my conspiracy theorist hat on and I love how this, the stories of these hackings, these rogue hackings come out because was it chatgpt, I think was one of the first ones to mention it. And right after they mentioned it, their stock shot up and they were like, oh yeah, I didn't know I had that kind of capability. That just made them all the more valuable. And then you heard Claude say the same thing and I think was the facehugger did the same thing. They're so willing to kind of share that they're systems are doing things that it not supposed to do. I find it curious that they're so willing to kind of share that. So I think in the next 18 months we will probably find out that a lot of the hype and a lot of the marketing of AI was more vaporware than real functionality that was touted. And whether that's going to be a byproduct of either overstating its abilities or B a necessary fallout of putting oversights in control of it, it's will probably be up for debate.
• 45:35
  Speaker C: So I'm guessing you're talking about that mythos 5 model that went out there and attacked a couple of things. Actually. I think mythos 5 is a Claude thing and it attacked hugging face and by going after developers and creating an email account, sending something to that developer, trying to get them to fish. Okay, Any other guesses?
• 46:01
  Speaker A: I. I think that's what it is. I think the reality of what AI is actually going to be is going to come into a more clear picture.
• 46:12
  Speaker C: Yeah, well, it seems to be doing that on the constant. Well, John, I truly appreciate your time and your experience. Thank you for sharing all of that with us. Truly appreciate your involvement with you've been heard dot com.
• 46:28
  Speaker A: I appreciate you guys having me on.