Latest / You've Been Heard / 344- How to Build Trust-Based IT Teams by Ian Johnson

Transcript

344- How to Build Trust-Based IT Teams by Ian Johnson · You've Been Heard

• 0:04
  Speaker A: Welcome back to today's episode of Dissecting popular IT Nerds. I'm your host, Doug Kamene, and today I'm talking with Ian Johnson, the head engineer of Full Stack at Bizkout. Welcome to the show, Ian.
• 0:15
  Speaker B: Thanks, Doug.
• 0:16
  Speaker A: So, Ian, like Bizkout, you're a startup, you're working in the startup space, but your history's not in the startup world here. Can you tell us a little bit about biscow, what you're doing now, and, well, anything you want to share, I'll just open the floor up to you.
• 0:31
  Speaker B: Yeah, thanks. Yeah, I think we could probably work backwards. So much like other technology trends, things sometimes become full circle, and my career has kind of done that as well. So I've worked in enterprise development, I've worked in fintech, I've worked in nonprofit, but I started in a smaller sort of implementation background with my own company. So right now what I'm doing is I'm working with Node JS and a small team of developers, and we're using Node Next, JS and a couple of other technologies to quickly deliver software for an aggregator, which is basically something like what you would see in Biz by Sell, or, you know, online marketplaces that have businesses for sale. So that. So the goal of our platform is to sort of put buyers in touch with sellers and then also have brokers in the mix somewhere so that they can make their buck off of whatever it might be. And so the founder is really prominent on social media, and so she's done a lot of things that bring people together to. To allow them to buy businesses. And so her goal is to make sure that people that, you know, the people that you wouldn't normally think would own a business, are capable of buying businesses. So that's the goal of the platform.
• 1:39
  Speaker A: So people like me, like, I could go buy a business. Not Bisco.
• 1:42
  Speaker B: Absolutely. And you could probably join our. Our Slack space and talk to her directly because she's very engaging. Or you could go through the site and look at the marketplace and put yourself out there as well. So that's. That's the goal of it, anyway. We've been working pretty hard to make sure that it's available to everybody. And there's been some speed bumps. Obviously, there's always going to be speed bumps when it comes to things like this.
• 2:05
  Speaker A: You have startup.
• 2:07
  Speaker B: Exactly. You have funding, you have timelines, you have people that are expecting certain things. So that's what I'm working on now. But if we were to wind the clock back and just Go all the way back. And just think about where all this started in Austin, Texas around 1997, when you and I were riding bikes. Right. Prior to that.
• 2:28
  Speaker A: Right, yeah. So. So for the listeners, Ian and I have known each other for a long time. We grew up in Austin, Texas together and we've known each other since we were like 12. But, but we don't always, you know, we live at different ends of the country and only speak very periodically nowadays. So it's great to have you on the show here.
• 2:44
  Speaker B: It's good to talk to you, Doug. And you know, it's. We even got in a fight on the sport court one time. I remember Doug's dad worked for IBM long ago and I had the privilege of knowing a few people that were into technology. That got me really started with computers back in those days. And so really what I was doing was chatting on AOL and Prodigy and those, those modem based systems back in the day and just trying to understand like where I was in the world and just, you know, growing up. And we were on, we were on modems, we were on dial up modems, right? So we had 2400 baud modems. And then I remember a distinct conversation with Doug at one point we were trying to send a file over a 2400 baud modem and he said something, man, if we just had that 14.4, we could just be different and it wouldn't take any time at all.
• 3:29
  Speaker A: The world would be our oyster, man.
• 3:31
  Speaker B: I don't remember what we were sending. It was probably some kind of exe that had Pamela Anderson. I don't remember what it was, but never know. But I do know that went back to Best buy, took the 2400 baud modem back in its packaging and returned it and got the 144 for the same price because they couldn't tell the difference. And it was a game changer. So we went from chatting on these boards, BBSs, things like that, to cable modem. And the first software that I wrote was a spam operation that essentially would log into AOL and find active users and send them email. And I made probably, I don't know, anywhere from 5,000 to $10,000 a month doing that for a period of time while I was younger. And it was a blessing and a curse because I could have probably started a legitimate business, but I was doing spam. And I think the money kind of sidetracked me away from what I could have been doing, which was building web applications. So this was a desktop application. Once time Warner shut us down. I went back into building web apps. And web apps was. I really enjoyed it. Just the idea of deploying to browsers. And if you think about your compile time on a desktop, you've got a runtime that runs on a desktop. Well, now you've got a runtime that's in distributed browsers and you've got cater to any number of them. Chrome, Netscape, all these different ones. And that to me was a real challenge. It's gotten a lot easier these days. There are packages and frameworks that allow you to do things very quickly and easily. But back then it was really a challenge. So got away from that and went into working sort of in E commerce. And I got my first development job, I think, in 2006 after all that settled down and I moved to Mississippi and I worked for company called, excuse me, company called fnc. We did loan origination software and loan servicing. And I went to, you know, interact with different lender banks, Wachovia, SunTrust, all that. And guess what? Right around that time was the big crash. And we saw all of those banks kind of fall off and disappear and also consolidate. So some of those brands, people don't really know that bank of America and a couple of the other ones were kind of. They had their hands in different things. And so when you talk about the CEOs and everyone's seen the movie with the CDOs and all that, you can see the technical side of these banks asking for requests that would literally just get thrown away. Because our contract with the banks was to make as many updates as they wanted. And if they wanted to throw the updates away, they could. There was no cost for that. So they were spitballing whatever they wanted at us. And it gave us, you know, the ability to code quickly, do what they wanted and just watch things that didn't work fall away. And to me, I think that was a huge value because there are many teams, many engineering teams that don't follow that same mantra, I guess. And what you'll see is, you know, they dictate specific requirements and it's a hard line and there's no flexibility. And what I really missed about that was the flexibility aspect of it. It's like if this doesn't work, it's just going to go away. So went through all of that, came back to Austin. I decided, look, I can't live in Mississippi forever. Look, I loved Oxford, Mississippi. Great. Beautiful girls, awesome place. Really cool tailgating. Sorry about the cursing there. I knew that was going to Happen.
• 6:39
  Speaker A: You know, that's fine. Our listeners will get over it.
• 6:42
  Speaker B: Well, hey, you know, it was, it was. It was a great time for me. And so came back in like 2010 and worked for another loan servicing company here in Austin owned by a guy named George Sow called Mortgage Dashboard. And we were operating as a startup. And I said, I talked about full circle, right? And this. So this whole, you know, I would consider that to be my baseline. Not necessarily the. The work in Mississippi, but started there, went from there to Volusion, which was one of my favorite places to work. Volusion is a E Commerce competitor for Shopify. Oh, yeah, we had store merchants. I learned a lot of things very, very quickly from the developers that I worked with there that came from Simi Valley, California. And we all worked in the same area. And, you know, you had situations where you would break something and become the hero to fix your own. Your own bugs, but then you had also situations where you needed to build something new. And, and so we produced something that had a patent. It was. I don't know if you're familiar with PCI compliance.
• 7:39
  Speaker A: Oh, yeah. So PCI'm on the board of a. Of a. The chair of the audit board for a large credit union that's based up here in our area. So, you know, PCI compliance and all those types of things, those are all part of the. Part and parcel of the day.
• 7:54
  Speaker C: Hey, guys, this is Phil Howard, founder of Dissecting Popular IT Nerds. I just want to take a few minutes to address something. It has become fairly apparent. I'm sure all of you will agree over the years, that slow vendor response, vendor response times, vendors in general, the. The average is mediocre. Support is mediocre. Mediocrity is the name of the game. Not only is this a risk to your network security, because I've seen vendors on numerous occasions share sensitive information, but there's also a direct correlation to your budget and your company's bottom line, not to mention the sales reps that are trying to sell you and your CEO and your CFO on a daily basis. That causes a whole nother realm of problems that we don't have time to address. Our back office program at Dissecting Popular IT Nerds, we've put together specifically for IT leadership. And it's on a mission to eliminate this mediocrity. And the best part is that we're doing this in a way that will not cost your IT department a dime. So if you'd like us to help you out, get better pricing, better support and jump on pressing issues in minutes, not days. Then contact us now so we can get on a call with you and conduct a value discovery session where we find out what you have, why you have it and where you want to go, and how we can improve your life, your IT department and your company's bottom line. What you're going to end up with is, number one, just faster support from partners who care about your organization's uptime and bottom line. And because you're going to be able to access our 1.2 billion in combined buying power, you'll be able to benefit significantly from historical data. And on top of that, you'll also benefit from the skills of hundreds of on demand experts that we have working behind the scenes that are all attached to our back office support program. So if you'd like, again, none of this is ever going to cost you a dime. At the very least, it's going to open your eyes to what's possible. Let our back office team provide you the high touch solutions and support that your IT team deserves so that you can stop calling 1-800-Gold-POUND SAND for support. Now, if you're wondering what does this apply to, this applies to your ISPs, your telecom providers, all your application providers, whether you're a Microsoft Shop or a Google shop, what you might be paying for aws, even Azure Colocation space, any of those vendors that you're paying a monthly bill to, we can help you with.
• 10:37
  Speaker A: Hey, it's Greg the Frenchman, secretly managing the podcast behind the curtain. To request a one on one call, contact us@internetopularit.net and remember, it will never cost you a dime.
• 10:50
  Speaker B: Okay, so you know about storing card information at rest and all those different things. And so we were violating one of their principles by storing card information at rest. It was secure to us, but it wasn't secure to them. And they came in like the mafia. And here's what's funny. You pay them a certain amount of money to, I guess, audit you and all they do is find you. They don't really. They can either approve you or take more money from you. At least at that time, that's what they did. So what we did was split the card numbers in half and we created a patent around this. So part of the card number would be stored in one database, the other part of the card number would be stored elsewhere and there would be a nonce or a ID that would form the card number at the time of checkout. And so since the one page checkout was what I owned as a developer, I saw that happen. And when the PCI people came and wanted to audit everything, they said where the card numbers were, one's, one's over here and one's over there. They're not stored at REST anymore because they're not card numbers, they're separate. Well, they've since changed all of that and I'm pretty sure that we probably can't get by by doing those sorts of things anymore. But we were coming up with ways to just, just, just move quickly and stay with the industry. So we did all of that integrated with like 10 different payment APIs, 10 top ones, and put a facade on top of it. I think it was chase payment tech, skipjack, authorize.net, a few of the other ones. And that was one of the biggest challenges I think I've ever faced as a developer. Just because, good God, those things are so bad. Those payment gateways are just really bad. Not spending too much time on that. Got away from working in the E commerce space and moved on. Went into nonprofit. I worked for the Michael and Susan Dell foundation for about three years directly with Michael and Susan Dell on their grant making system. They were two of the major logins on our system. So we always had to have uptime, we had to make sure that we had automation and availability. And when, whenever we saw one of the golden users, which was Michael or Susan Dell, pop into the system, we got alerts and it was okay, they're in there in the goods.
• 12:50
  Speaker A: Everybody, all the, all the screens come up and everybody's making sure that this system is running exactly running like a top.
• 12:56
  Speaker B: And it's funny because I remember one specific case where Susan had gone to Hawaii and she was on a, she was on a machine that hadn't been updated in like five years because it was on some property that they didn't ever go to. And it had an old, old version, version of the browser. And so when she logged into the grandmaking system, it didn't work properly. So they have a tech team that's part of the family and then the next layer is us and then the next layer of that is my management and it comes on all the way down to me. So I, I would get like this game of Telephone where it's like, hey, Susan can't log into to Internet Explorer. It's like, okay, well what version of Internet Explorer is it? And I would wait and it would five layers of people and then eventually it would come back. It looks like it's IE6. And I'd be like, well, that's your problem. And so it would go back, and I would have to wait and wait. And finally we figured out that it was just a really, really old device and that we had to send someone from her personal tech team to go update everything. And once they did it, it worked. But that was the kind of. And this was at like 5 in the morning, by the way. I don't know what they were, why it was five in the morning, what they were doing, but we had to have full availability. So I built a lot of automation around. Like, since we had a small team, I would take the production database and make sure it was updated down into low environments via a lot of scripting and things like that. And so working with a small team like that, you want to have a replica of your prod environments that you can actually code against it. And I learned a lot about making sure that I had a mirror from my production environment down to my lower environments. And every morning when something new showed up, I would just run a few tests and I had unit tests and things, and I would smoke things out by myself. Just like, okay, we released a prod, and we've got a little bit of an issue. I don't know how that got by our test before, but now we found it and let's fix it. So I learned a lot about that in that regard and became a lead there, ended up leaving and went to Charles Schwab. Charles Schwab. Let's just. I just want to make this a short conversation. Great place to work, but you got to be on the right team. That was the place where you learned that one little mistake can cost hundreds of thousands of dollars in no time at all. Hundreds of dollars, if not millions. And so I learned to work very, very quickly, work with product. And I learned the value of what product wants to have delivered versus what you're trying to do as an engineer. Because sometimes you can make an engineering environment your, I guess, mad scientist layer. You can start to do things that you think are something fun for you to do, but doesn't necessarily translate to what the business wants. And. And so that's where I really learned, like, quick, quick, fast and efficient is the best way to do measure twice, cut once. And that's where I learned that. And I went from senior staff engineer to lead engineer. I actually left and went to Dell for a period of time. They brought me back. They said, the SAC is falling apart. Please come back. I came back and then became a director. And once I became a Director at Schwab, I started to learn another valuable lesson. And that was when you take your hands off the code and you take your hands off of anything.
• 15:54
  Speaker A: Yep.
• 15:54
  Speaker B: You start to become more of a voice than an influence. And it's harder to. What I always say is like, teach a man to fish. Right. You know what I mean? Don't, don't tell them, you know, teach a man to fish. And, and I had a real problem with that once I got to that level.
• 16:11
  Speaker A: And I felt that's a hard transition to make. Like is, is people move. Yeah. When you move between one, your space of, of being a hands on leader to being, you know, a leader of influence. You know, you say you, you have a voice, but you don't necessarily have hands on the keyboard anymore in the same way. So you, it's like you have to be able to effectively lobby the people who have their hands on the keyboards and make them want to do the thing that you want them to do. I mean, you can always order them to do it, but that may not work out for in your favor either.
• 16:42
  Speaker B: That's absolutely right. And where, where do you draw the line? Right. You can't micromanage people. First of all, I was an ic, so I wasn't a manager, I was an IC the whole time. You can show them what you think that things should, should, how things should turn out or what they, what you think they should do. Or you can even make a group decision. Now what happens when a week later the group decision that you thought you made was not the same thing and they implement something different. So you come back and you say, I thought we decided on this. And it's all written down plainly in this document that we all agreed on. And you wrote something else. Now what do you do in that scenario? The time has already been wasted. You've already trusted the people around you that, that you think should be implementing the, the software that is good for the business, the stockholders, the shareholders, everybody that you know, anybody that is, has a stake in this and it's, it's painful to watch sometimes. Now on good teams that doesn't happen. And I think good engineering teams are very, very rare because there's one factor, there's one factor and it's just trust and respect of each other. It's just, you just have to trust each other. And if you can't trust your team, you know, there's any level of micromanagement is going to come death, death. 5,000 cuts and the team will disintegrate and I've seen that happen as well. I saw it happen to Schwab, I saw it happen after that. I saw it specifically happen at Q2 software. So. So after I left Schwab, I went to Q2 Q2, I was the head, what's called ATG, which was an advanced technology group and reporting to directors and SVPs and all I worked with was engineers because we were like the SWAT team of different, so different teams on Helix. Helix did 15 million transactions a day with Visa and ran mostly on a SQL Server database platform with some NET core and we were doing some containerization and trying to rewrite a jobs platform which I had a lot of input on and hands on input on. But the hardest thing to deal with there was the engineers. And I hadn't seen it like I had a lot of trust at Chuang, varying levels of trust, but you know what I'm saying, when I got to Q2 as a newer person, I don't know what it is. What do you have to do to prove to these people, like do we have to go out and beat each other with sticks to prove that we're actually, you know, worthy of our positions? Because that's what ended up happening at that place. We started to kind of fall into this, like, I don't trust you, I don't know you this and that. Everybody was pretty new and for lack of a better word, we were measuring things. So anyhow, that being said, I couldn't, I couldn't handle it. It was two years, I had hundreds of thousands of dollars in stock. I had a good financial position. Other people were really golden, handcuffed the same way I was. I could tell they were miserable, everybody was miserable. And I said, I gotta get outta here. And so I went to this startup. So that brings me back to the.
• 19:30
  Speaker A: Full circle, back in the startup space.
• 19:32
  Speaker B: Back to the startup space. I said, you know what? This is a place where I can leverage my influence and be heard and be seen and actually just have hands on, no blockers, get things out quick and solve problems.
• 19:45
  Speaker A: So there's a couple things I think about just. And I appreciate you sharing that journey. So a couple things that came to mind for me. One is you talked about the teams and how the differences in teams, the effectiveness of teams and in my, where I work, we just went through an exercise. We have an executive coach who helps us and kind of helps steer us for the last year or so, which has really been helpful for us. But one of the things, the exercises that we did recently was we walked through rating and understanding the ideas of, I guess I call it the team performance curve. So this is a pretty common chart where, and of course we're on an audio podcast, so I'm just going to have to verbally describe it. But on your, your bottom axis you've got the, the effectiveness of the team and then the vertical axis is the performance impact of the team. So you start starting on the left. You start with like a working group and as the team effectiveness declines, it's like a pseudo team. And then as it starts kind of rounding that bend and you know, coalescing, it becomes a potential team. But only the performance impact is going up and you get to a real, what they call a real team and then finally a high performing team. And to me it sounds like what you've experienced is all aspects of that graph you've been on places where it's a pseudo team, you're a team in name only and the people are, they're not in it as a group. They, they're in it for themselves. And there's a, there's a lot of fighting and challenges going on and stuff like that. But you've seen, you've experienced the other side of that, of being on either a real team or a high performing team where you're really seeing the difference. And the biggest challenge in that space that I see is that once you've been on one of those high performing teams, it hurts so much more to be on the pseudo team.
• 21:31
  Speaker B: That's absolutely right. That's why it's so painful when you, when you see it happen again and again. So you don't want to be the chicken little. You don't want to be the one that says the sky's bumbling in front of people and saying what you're doing is wrong, it's going to fail. But there are a lot of people out there, including managers, especially managers, that don't listen to that type of input and they just wait to fail and they'll actually learn from their own mistakes. So you can say that the decisions we're making right now are going to be a problem and you try to fix it. Well, they don't know that you're actually trying to help them. They just think that you're someone that's calling out issues that may not exist. And then once they realize that you were correct, now they have some kind of a, a vendetta against you because you're the one that called it out before and didn't do anything about it. But it's not that I didn't do anything about it. You're the one that's making the decisions. I'm trying to give you the input to make those decisions. So it's a really tough balance. I think that, you know, what you're getting at is like, you know what, what does it take to build a high performing team? And I think I watched a video with Steve Jobs not too long ago and he said that his, you know, way back and way before he was, was even older. He had hair and everything. And he said, he said that, he said that his biggest, his biggest job role was recruiting because he needed to figure out how to get, you know, the right people with the right other people to build the correct teams. And this is something that I think is lost in today's industry. I think that like, you know, a lot of times this is overlooked and it's not something that usually happens with planning. It happens naturally. Like sometimes it just happens and you have this great team and sometimes it, you just don't. And what, you know, what, what forms that I don't know all of the tangible details but I do know that the main thing is trust and being able to trust your coworkers and not. You don't play political games. You don't do any of the things that are not part of an objective engineering team. You, you literally just try to get the jobs done and you eat your crow whenever you, you have an issue. Like if you're wrong about something, just be wrong about something. Don't take it all the way to the to as long as you want to take it, just be humble. And sometimes that results in fast moving delivery. And fast moving delivery is what everyone wants. I don't care who you ask on any engineering team, they want the software out quickly and they want it to work. Those are the two things that everyone wants. It needs to function and it needs to exist. It doesn't exist if you can't release it. So, so for me, I've seen managers stand in the way of that. I've seen project managers stand in the way of that. I've even seen Scrum masters stand in the way of that. People need to loosen the grip a little bit from every different role and every perspective on these teams. And if they can, you loosen the grip and then you're relying on people's skill sets. Now that's where things become a little bit hairy because if you have a bunch of green people on your development team and you trust them too much, they over deliver, they Break things in prod and you're going, why did I, why did I trust this person? I don't know why I did that. Well, you have to, that's, that's the answer. You have to trust your team. So build the right teams, that's all. Now how do you do that? Well then it becomes even more complicated. Pay grades are so high now. I mean, think about what it takes to, to, to, to employ a top notch engineer. A lot of people now I'm, I'm reading posts on LinkedIn, you know. Oh, you know, the, the, the junior devs are better than the senior devs ever were. The junior devs now are awesome. They're using Chat GPT and also it's like, okay, fine. One thing I've noticed, and this is not a knock on junior developers at all is problem solving capability of senior engineers is way beyond the problem solving capability of lower end engineers. Yes, they can develop software quickly.
• 25:07
  Speaker A: The skill isn't in the code. The skill is in the ability to write the code.
• 25:11
  Speaker B: And look, being an old dog, I don't want to be the person that's like, but look, if I pull up a notepad and write a JavaScript closure for you and ask you why this is a closure, you better be able to tell me quickly without opening ChatGPT. Okay, that being said, if I say, hey man, I have a high level problem, we need to integrate with Braintree and process the payment and I need you to add some web hooks so that we can process the responses from the payment processor. Can you please do that and just create a prototype? I don't want to see a separate application with 35 webhooks. I want to see a direct integration and one endpoint so that we can actually see how this works. The, the, the what I've seen and this is just me and maybe I'm wrong about this, I'm not saying I'm correct about all this, but junior developers tend to over engineer things and you have to watch them so much more closely because they don't want to ask questions. They, they're afraid that if they ask questions they get into the imposter syndrome thing. And then after that it's like, oh well, he thinks I don't know what I'm doing and I never do that. I always, like I said, I err on the side of trust, so I don't know if I'm down a pit.
• 26:13
  Speaker A: That's okay. So I'm glad you brought up imposter syndrome because like, so as a Leader, How. How do you help your teams overcome some of those blockers like imposter syndrome? I'll focus on imposter syndrome to start with there. Like, you've got team members that got imposter syndrome. You've been in this. You've been in lots of different fields here. How you help your teammates, especially the junior ones, overcome that and feel confident in their space, That's a great question.
• 26:38
  Speaker B: It's not really difficult. What you have to do is create some rapport with them and let them own their successes and their failures. If you start to persecute someone for failures that you perceive on your side that aren't necessarily failures or whatever they might be, you're actually becoming a negative influence on their career. Because when they do things that are correct, they don't hear anything. When they do things that are wrong, they hear all this negative input. So you have to kind of do both sides of that equation. There's successes and failures. Don't steal people's credibility or credit for things that they do right. Always call them out and give them, I would say, give them props for what they've done correctly so that they know they're doing the right thing. But if something negative happens, pull them aside and say, hey, look, you should have ran more tests. And look, I had to fix this in production. I have a great example that I just did this yesterday with one of our contractors. He didn't put a null check on string length check and broke our entire profile page in production for one simple call that was basically to see if you had discount codes on your account. And I could have just eaten that and not told him, but I had to bring him aside and just came like, hey, man, why didn't you run this one use case? He said, I thought I did. I thought I ran that. Okay, well, I'm glad that you thought you did that. But I did it, and it was in production. It was broken for about 15 minutes, and users probably saw it. So, you know, I had to fix it myself. What are we going to do to fix it? And he came up with a good plan to fix it. He came up with some tests and some other things. So positive outcomes can come from negative, I guess, reactions. I don't know if that makes sense.
• 28:05
  Speaker A: Yeah, and I'm thinking about. So I, you know, I work in the IT operations delivery space. You know, I'm a technologist in the, In. In delivery of things. You know, network switches, gear servers, that type of stuff. But I don't develop. So you're in the development space. And, and so I'm thinking about some of the differences in leadership and how you have to lead people who are developers that in it's, I don't want to call it like totally different skill sets because leadership is still leadership. But, but some of the tactics and the things that you put forward, I think there's some really important lessons and valuable lessons there, like what you just shared. You pulled aside this gentleman and you highlighted to him where the gaps were, but you asked him for a plan. So like I always describe this as giving an exit. Like you give somebody an exit ramp, for lack of a better way to put it, to satisfy you or please you. So if I'm the leader and I'm going to say I'm going to offer some sort of critique, I make it pretty evident what the path is to get back into. Like, like I would call it, you know, whether it's to impress me or anything else like that. And it's not that they need to like kiss the ring, it's just that that's the like the human nature of it. Like if I, if I got criticized for doing something, I want to know how do I get back into the good graces with that individual to know that I'm, I'm on a good place. And that goes a long way towards building the rapport and the understanding and the trust that you've highlighted as being so important in so many of those teams.
• 29:31
  Speaker B: Yeah, I mean I think it's, it's simple but it's, it's also complex. Right. This is not an exact science and it's not something that everyone is really good at. But you know, being, being a human being and not being a robot as, as a person that is a leader in the space is very, very important. At the end of the day we all check out at 5pm or 6pm and we go do our own thing. We have a life outside of this space. It's not, you know, unless, unless you're working with pacemakers or something, it, no one's going to die if you implement a bug or you send a bug to prod. So let's just find a way to be better. And so a lot of times just finding a progressive way to be better and not being super negative and just having these histrionic fits about certain things is very, very important. You have to be the level headed person, you have to be the one that doesn't freak out when there's a problem. And if prod is down and everyone is Running around their hair on fire. You need to be the one that goes into the database and runs some queries and figure out, figures out what's going on and then brings it back to the team. So leadership, hands on technical skills usually go hand in hand with that level headedness. I've seen some managers that are people, managers that don't have that acumen and they literally will sometimes lean on certain SMEs and let's just talk about management for a moment. I think that management can go wrong a lot of different ways. And one of those ways is when they and I talk about trust a lot. But if they trust the wrong people based on a lack of skill set. So if. So if they really don't know how to solve specific problems or haven't had the experience in solving specific types of problems, they might trust the wrong people. And when they trust the wrong people, that is a detriment to an entire organization because those people become propped up artificially and they make decisions that are bad for everyone. And everyone knows it, but they can't say anything. And this is one of the bigger questions, I think, how do you. And I've had problems with this before. I don't like to jump rank. I don't want to go up and complain to like an SVP over something. I usually like to just do my job and make sure that everyone under me is happy, all of that. But sometimes you feel like you need to jump rank and call somebody out. Now how do you do that? When someone has more authority than you, higher pay grade than you, and you know they're doing something wrong, what do you do? I mean that's, that's a real question that I've struggled with. You pull them aside and talk to them in a, in a, in a vacuum, you don't want to call them out in front of the team. That's unprofessional. And to me, even talking to them about these things can be unprofessional because maybe there's no fix. You know what I mean? So, so that's, that's something that I've, that's a question that remains in my mind that I've dealt with several times. Some, some of these leaders that are higher, higher up, I'm talking about founders, svp, we're talking about the upper echelon C suite people. They can almost feel like they're not approachable at all. And I just wish that some of them would come down to the team level, make themselves approachable and say, how do we fix. I know we've got some issues. I don't know where the issues are. How do we fix it? They rarely do that. They're usually up on their, on their pedestal somewhere, and they don't do that. And I encountered that at Schwab. Charles Schwab has a real problem with that. SVPs have no clue what, what the teams are doing, and they trust a couple of people to tell them. So that's another subject. We'll leave that one as kind of an open. Yeah.
• 32:52
  Speaker A: Okay. Yeah. No, and it's true now. So I don't. I, you know, I'm a, I'm a C suite executive, but I work in a small nonprofit, so, like, I'm, you know, I'm, you know, we're 500 staff, not 50,000. So, like, it looks different. I will say that at least in my space, I try hard to make sure that I'm accessible. But it's a lot easier when there's 500 people in the organization to feel that way than there is when there's 50,000. And that's different. But, you know, I'm on the board of. I mentioned I'm on the board of a credit union, and they have about a thousand employees, and they have a very great work culture. So this is not an explicit knock, but even the jump between one to the other and the difference in industry, there's, there's, there's absolutely a stratification there where, like, the C suite executives and like, the line staff are. Are much further disconnected just by nature of the business, you know, so, like, that. That can play a part. I, I did want to, I wanted to pivot slightly. You mentioned, just kind of offhand. But you talked about junior developers. This isn't a question about junior developers specifically, but you mentioned about junior developers, like, potentially using, say, like ChatGPT and some other tools to do this in your space. So, of course, this is the talk of the moment. You know, all the AI tools and all the things that are coming down the line. I've been exploring this in, In. In my space and in the nonprofit space. But for you, there's, there's some really, I would argue, compelling support cases for some of these AI tools. But I'm just curious what your feelings are on them. How. How have you been exploring them? And what do you, what are your thoughts?
• 34:22
  Speaker B: Yeah, thanks for that question. Because it's come up. It comes up constantly. And so you have two different schools of thought on this. You have the people that are actively developing software with their hands and other people who want the software delivered quickly and don't have any idea what you're doing when you're digging the ditches, right? So the people that, the people that are not hands on are the ones that really love this idea of Chad, GPT and AI. Like, hey, eventually we don't even need developers. I can just tell them to do something. Developers are a pain in the ass because all these technical details come out every time I ask them something. Then you have the developers who really want to use these tools. There's nobody that I know that doesn't want to use them. The problem that we have is quality of what comes out of those engines. We had one at Q2 that was based on our internal software and it was trying to suggest things for us to do. And we ran into one specific. And I'll ask you this question. I think probably you're going to be able to identify it very quickly. If you got a legacy code base, you got a new code base, you unleash an AI on all of it and it's making suggestions. How does it know the difference between crappy code and good code?
• 35:26
  Speaker A: Right? Or does it hallucinate or create some sort of hybrid between the two? Where it's like, right, like some of this is JavaScript and some of it is like, you know, RPG convert all the AS400 and you can't dump that.
• 35:40
  Speaker B: Sometimes it'll reach out to the open market, which is the main code base that they leverage, and you'll get things like entity framework that you're not even using. It'll be a suggestion. So we did a pilot test on this internally at Q2, and we found that from day to day, the suggestions that this AI was making were completely different. So one day it would be, here's a suggestion for this code block and you would go back to the exact same code block the next day and it would give you a different suggestion. And to me, right then and there, that's the main quality problem. Per my understanding, the human input needs to be relative to what you do. Like which, say you're on a Reddit thread and you're uploading and you're downloading things. You need to be able to upvote specific good answers and downvote the bad ones. And I don't, excuse me, I don't think the learning aspect is there for a lot of these LLMs. They're really good at taking a question and digesting it and producing some LLM, some language back, but they're not great at being consistent and actually giving you valuable data. They're just giving you data that's in their database somewhere. So if we could improve on any of these tools, it would be more human input about quality. We actually got rid of that AI at Q2 because of this reason. Well, there was another reason. Some of the upper brass was a little bit scared that it had too much access to some things like kubernetes, clusters and things like that, and didn't like that at all. But it was just logs. We wanted to have access to all of our logs. The idea was that mainly it was a code base tool and a log tool. So if you wanted to digest logs, you could do that, which is great. They're really good at digesting logs. In fact, as that's probably the main use case for an AI that I know of, that's very, very valuable. Give me the log for the last 10 days on these 10 different machines, only SQL Server, and give me all the event loops and any type of stack traces that you see. Boom. They'll give it to you. That's perfect. Then you digest that data. But if you say, give me a ORM layer for this application that needs to connect to MongoDB but also be swappable for Postgres and even have SQL Server in the background for another different use case, the thing just on itself because there's too many different things. And I apologize for the language, but that's what happens.
• 37:49
  Speaker A: And so for me, passionate. That's. That's all that's. We're passionate about this one. You know, it is because I would.
• 37:55
  Speaker B: Love to trust these things, but I don't. And I can't object. You just can't yet. Now, someday they will be better. Someday they can actually solve problems. And someday they are going to be capable of writing small applications. But yeah, they're going to be capable of leveraging senior engineer experience. There's always going to be a human being pushing buttons to put pieces together. And. And that's where I'd like to be, you know. You know, I love, I love writing software. I love the craft. I love being able to make things simpler and make them as simple as they can, but not more simple. But. But sometimes, man, there's boilerplate things that you've done before and you're just like, I wrote this HTTP client class like a year ago and I remember doing this, but I don't remember all the details. Do I have to start over? Where did I put that code? I don't know. Where I put it, you could push a button on the AI and say, you wrote this a year ago and here's your block of code and started here. Thank you.
• 38:44
  Speaker A: Even Jordy laforge. There had to be Jordy laforge to run the Enterprise right in front of all that technology.
• 38:52
  Speaker B: No, push the buttons. And even Data still had to push buttons. And a captain would know that.
• 38:58
  Speaker A: So I appreciate your thoughts on that. I think that was insightful from like an alternative perspective. Well, I don't say alternative to me because I work in just in the infrastructure space. So it's a little different to think about how it impacts in the development space. Another thing I was so just maybe more fun, humorous discussion points and things to talk about. So have you. Have you ever. You tell us the story? One that's good, safe to share. I guess I would say a time that you've done something like in your professional world where say, like, something went wrong and you're like, oh, crap, like, we did it wrong. And like, it was almost embarrassing. But you get to laugh about it now. And I usually start with a little bit of an example. So way back in the. Way back in the day, it's funny now. It's like, I guess it was almost 15 years ago now, maybe a little more. I was at a customer site replacing some servers and their server room was really, you know, just 1980s, like classic server room. Like wood paneling and wires on the floor, no, you know, nothing underneath and stuff. So behind their racks with just stacks and stacks and stacks of wires and you had to step on the bundles of wires to get the stuff in. So I had to tiptoe through it. I'd done this many times, but one time I did it and the bundles of wires, the one rubber pulled on the next rubber and the pulls on the next one. It had unplugged the expansion chassis for their AS400. Here I am, the network admin and I are in there working and all of a sudden the AS400 guy comes in and he's like, what the hell's going on? What the hell did you guys do? We're like, we didn't do anything. What are you talking about? And what ended up happening was the plug had come out. So we look and we're like, nothing's unplugged. It all looks fine. And the plug had been pulled just enough to disconnect it, but not enough to make it fall out of the outlet. So we couldn't see at first that it was done. So, like And I was a. I was a contractor here, you know, so like here I just took out their. You know, they're a manufacturer. Like, they could probably quantify how much money that that mistake cost. I don't. I never got to know that number myself. But. But yeah, so yeah, like open the floor.
• 41:04
  Speaker B: Yeah, there's two that come to mind, but the one that really we. Are we almost done? Oh, yeah, yeah.
• 41:09
  Speaker A: You got a little while. We still got a few minutes.
• 41:11
  Speaker B: Okay. So the one that comes to mind is I worked in a data center too, called Data foundry, which was texas.net back in the day. And that was downtown Austin in the old building. So I worked the night shift. I would come in at 8pm sometimes on a Friday. And everybody was going downtown to party and I was going to work. And so I ended up in this building in the state center all night. And we had biometric scanners and so all this cool tech. And so every once in a while somebody would get lost. It was under the Museum of Art in downtown Austin. Someone would get lost and try to put their hand on the biometric scanners. And I'm gonna give you two little. This is kind of funny, but I'm gonna go to what you were, what you were asking. After that, this guy came by with his girlfriend. And the biometric scanners only work for us and a couple other people, people that access to the servers. He goes in there and stands up real straight. I'm looking at the camera, I can only see people down there. And we're talking about. It was an old bank vault. So the old bank vault door was this thick, two foot steel, automated, had a man trap. And if you opened it, you would see a fluorescent sign that said Data Foundry. Very, very ominous. Like if it opened, it was like, what in the hell is this thing? Right? So this guy puts his hand on it and nothing happens. So he takes his hand off, his girlfriend comes up, puts her hand on it, and I push the button to open the door real straight. The door opens and he looks in there and sees that fluorescent sign. And I'm sitting there just laughing because I'm the only one in there at the time. And the door shuts automatically because you can't get through the man trap anyway. And it's not a security issue. And so door shuts. He puts his hand on it again. Nothing happens.
• 42:43
  Speaker A: Nothing happens.
• 42:44
  Speaker B: She put hand on it again. It opens up again, again. And he looks at her like, who are you? Right? Like you have access to this and I don't what in the hell is this? Right? And I come on the speaker, can I help you? And he's like, we're good, we're fine. And he just. They just. They just bug out. But when I worked in that place, we had a halon system. And the halon system had sensors on the floor, and the sensors would sense moisture because the machines, obviously no moisture was allowed by the machines. And this was a big deal. People paid a lot of money to be in that, that colocation. And one of the guys went out there and he's. The larger guy had been hired and he was. It was hot day. And he went out there, he was sweaty. And I was thinking in my head, I was like, man, I hope he doesn't. Hope he doesn't sweat on the, on the tiles out there. You know, it could cause a problem. He goes out there, the halon system goes off and just covers the whole room with.
• 43:35
  Speaker A: No way. Yeah.
• 43:36
  Speaker B: And I couldn't find him. So I'm in, I'm in the op center and I'm looking out there and I'm like, I can't even see the dude. And he finally came walking back with his hands out like this and made his way back. So that was one thing that's similar to what, what you said, but failures in software, the. The worst one was at Schwab or trade Blotter, one time failed, and it was out of sync with the open market. So we have, as a trade blotter, you put trades in on our system. Schwabadvisor center has the ability to make trades. And so hundreds of thousands of trades were in this blotter about to go out to the open market while it failed and it couldn't be processed. Turns out the trades were a net positive for the people making those trades because they're good, they're good traders. I mean, they operate on this platform. And we lost in 30 minutes close to a million dollars just because that trade blotter was behind. And once it went out to the market, it was a net loss, but it should have been a net positive. Wow. Yeah. And so they have insurance that, that, that, that is made for. For fixing that. But this was a very simple software bug. It was something that was had to do with mapping fields, and the fields weren't mapped properly to go from one step to another. So every time it tried to move forward in production, it literally just couldn't map a set of fields to a shorter set of fields that wasn't properly. And so it would fail, but it was a silent failure because it was very old. We also had mainframes, so we had these mainframe services that nobody touched. Usually I think they were written in cabal or something. And, and so once it got to that layer, there's no testing it until it gets to the live market. How do you test a live market? You can't replicate a live market down into lower environments. You just have to make sure that it works when it gets there. So that was a big learning for us. Then another one, another big failure that. So you talked about. So that was one you talked about. How do you, how did you fix it? Right. So what were the two questions? One was what was the failure and what was.
• 45:19
  Speaker A: Oh, I just, just in general, just. I, I was just interested, mostly interested. Just the hubris stories of that. We all have like these war stories where, where we did something, we did something like boneheaded and we're like, oh, that, that cost a lot of money or this thing went down or you know, know.
• 45:33
  Speaker B: Well, we thought we were fired at that point because of that, but it turns out they had insurance backing that was like, guys, it's not a big deal. Just don't do it again. Please don't ever do that. It's been a while since we had to dip into this pot to pay for some stuff like this, but. And then the other thing was when I was at volusion, we had 40,000 store merchants and I don't know if you're familiar with jQuery, but jQuery has a. So jQuery for those who aren't familiar with it, is a quote, cross, cross platform across browser library that allows you to write code that works on pretty much every, every browser. Back in the day, the event structure was a little bit different, so they encapsulated into this little package that everyone used. And so back then I think it was like jQuery 3 or just before jQuery 3, maybe 2.1. We were building some things for all of the store merchants on the core platform and I was writing the code and I saw this dollar sign. Well, the dollar sign is shorthand for the main jQuery object in most, in most environments use a dollar sign. Okay, well it turns out there's another JavaScript library called prototype. Prototype also uses the dollar sign. What we didn't know or what I didn't know is that there were some people that were skinning our storefront as a third party, I guess, offering and being paid to do so and using Prototype. Well, prototype use the dollar sign. Well, it turns out that was like 30% of our stores, and I didn't even know it. I decided, you know what? I'm going to change this dollar sign, make it more explicit. We're going to move it back out and just make it say jquery. No, it was the other way around. I'm going to take jquery and turn it into the dollar sign. Well, the dollar sign overlapped with prototype. Didn't break any of our tests. Didn't break anything. Went all the way out to prod, and I broke 30 of the stores, and we're talking about 40,000 merchants. As soon as that goes out to prod, the system goes nuts. Support calls. Everybody's freaking out. What the hell is broken? What happened? And I'm going, oh, damn. I was like, was that me? That can't be me. It couldn't be me.
• 47:25
  Speaker A: Why?
• 47:25
  Speaker B: And there are why?
• 47:27
  Speaker A: Dollar sign.
• 47:28
  Speaker B: Yeah, I just changed from literally typing out jquery to a dollar sign for shorthand to make things easier to read or just shorthand. And it broke everything. So I went down to one of the sites and I gave me an example. I was like, I need. I need an example. Like, figure this out. It was one of the sites. It was an airsoft company. One of our biggest sites was an airsoft sales company that sold these airsoft guns. And it was just completely. Just borked, right? I go in there, look at the source, and I was like, oh, man. I was like, okay, I see it. I see it. Prototype. Prototype. Where's prototype? Why are we at prototype? Who's got prototype? Well, the third party they pay for has prototype. Okay, fine, whatever. One quick release, and I fixed it. It took no time at all. We weren't down for very long. I just rolled back what I did and pushed it back out to prod. And I started receiving things from these. From these merchants. Like, I was. I got a wine and cheese basket. I got all these other things, and I broke it. I fixed it, but I became the hero still.
• 48:22
  Speaker A: That's great.
• 48:22
  Speaker B: That's awesome. And the dev team was like, what did you do? I'm like, I broke everything. That's what I did. I just fixed it.
• 48:29
  Speaker A: So you're like that. I don't mean to compare you directly to it. Arson is like the arsonist dude that puts out the fire. You know, it comes back later.
• 48:38
  Speaker B: So I learned right away that, like, yeah, it's not always as big of a deal you think it is, but. But, yeah, that was one of my biggest memories. At that place.
• 48:46
  Speaker A: Nice. So we're coming to. We're coming up to the end of the podcast here. Just got a few minutes left. I, I always. I, I'm going to ask this question to you more explicitly because we have a shared history in this particular time frame. So you. We're both technology people, but what was your first technology experience, Your first computer experience? Like, how did you get into being in space?
• 49:08
  Speaker B: I had a 286 IBM that my mom bought me. And I think it had something to do with you guys talking about IBM computers all the time, because I'm pretty sure you had. Before I did, it was a 286 that had a. It was flat and it had a little flip thing on it. Before that was all video games for me. Yeah. Into that thing. I started going through everything. I started going through every file in the system. There used to be something called autoexec Bat that was on Windows machines. Started the entire machine and I would go through every, every executable and run it. I would run everything that I had and see what it did. And I ran that auto exec. I said, it doesn't do anything.
• 49:43
  Speaker A: And I deleted it.
• 49:45
  Speaker B: I tried to restart the machine and the machine wouldn't do anything. The OS just completely paused and broke. And I had to call my friend Rio Marc Antonio at the time, he had another machine just like it. I said, what did I do? He said, what, did you delete us? I deleted the autoexec bat. My mom's going to kill me. Can you figure out what I did? He said, don't delete that. That's the bootstrap for the whole os. Well, I didn't know that. And that was, that was probably the moment that I realized, like, don't go around screwing with it too much until you really understand how things work. And then once you understand how things work, you can make surgical changes. And that's how I approach software with legacy code bases now because of that trauma. So that machine and I had that machine in that apartment in, In Spicewood Springs. You and I used to chat back in the day. That was, that was the machine. And then I had a. Later on I had a Packard Bell 386 and, you know, the rest is history.
• 50:36
  Speaker A: Yeah, Yeah. I remember when, like when my dad bought like that IBM PS1 that had had the 2400 baud modem and it was like a big deal that we all got on AOL and Prodigy and like we were sticking around in Nightfield, like, we're gonna Go on this. But at the time, we didn't, it didn't dawn on us how expensive it was. Remember when it was like 6.95 an hour? And we were so that was one.
• 51:00
  Speaker B: Of the biggest scams is that they were charging for the minute. And AOL and Prodigy, you know, you know, the main users of those services were children. At that time it was, it's obvious that most of the kids were so curious about these online services that they were on there, using them. And I, I got my ass chewed multiple times for huge bills.
• 51:22
  Speaker A: And that's the story of the late Gen X, early millennial kids, right?
• 51:26
  Speaker B: Like it was. My mom used to call it a glorified walkie talkie. Right? Walkie talkie. And occasionally she would pick up the phone and start talking and then the model, you know. But then I would hear my mom from the PC speaker. Hello, Hello. I'd be playing Doom or something, trying to shoot someone. I'm like, what are you doing? And I said, go run out. What are you doing? You know. And luckily none of that ever happens anymore. I've got a gigabit feed into this house and as much Internet as I could ever want, so.
• 51:55
  Speaker A: Right. How things have changed. So, so, so last, last thoughts here. Then we got to, we got to wrap up. But for, for the listeners out there, what leadership advice do you have for people that are coming up in the space? You know, whether it's being, in your case, being a development leader, how, you know, what, what advice do you have to share with the folks that are getting into their first leadership roles and what things they need to pay attention to?
• 52:17
  Speaker B: I would, I would say, and this is probably the first thing that comes to mind is, is aptly, aptly, okay, respect your engineers and your SMEs. Find, find the ones that really are your key, your, your, your keystone to your environment and make sure that you trust and leverage those individuals. And never get yourself into a position where you are undermining decisions from people that you trust. As an SME, establish your SMEs, your subject matter experts in different areas and make sure you're leveraging them and respecting them and not undermining their decisions. One of the biggest problems I see with management is that they think that they are the end all, be all, and that engineers don't matter and that whatever they say is secondary to their decision making skills. The best management and the best leadership leverages the input from their engineers and their subject matter experts and makes decisions based on that. That's all you can ever do. And don't ever believe that it's you, that that is the end all, be all. You have to leverage your. Your engineers. That's what I would say. And that goes for every engineering organ. That's some. That's one of the biggest gaps I've ever seen. I've seen great leaders do that. And you loosen the grip and everyone. The stress level goes down, the organizations run like butter, and you just don't have to worry about that much. But if you have anxiety about talking to your leadership and about them changing your decisions and changing your input and, and not listening to you, that means you're not valuable as a, as a, as an employee or, or a professional. And no one wants to feel that way. No one should feel that way. You hire people based on what they can do and you trust them after that. And that's, that's my perspective on it. Thanks.
• 53:51
  Speaker A: Awesome. I appreciate that insight. So, Ian, thank you so much for investing your time with us on the podcast today.
• 53:57
  Speaker B: Well, thank you, Doug. It's been really good talking to you, man, and we should stay in touch.
• 54:00
  Speaker A: Absolutely. So that's a wrap on today's episode of Dissecting Popular IT Nerds. I'm Doug Kameet, and we look forward to coming to you on our next episode.