Latest / The MonkCast / Why ChatGPT Recommends Your Competitor: LLMs & the Future of DevTool Marketing with Adam DuVander

Transcript

Why ChatGPT Recommends Your Competitor: LLMs & the Future of DevTool Marketing with Adam DuVander · The MonkCast

• speaker-0: Hello there, Kate Holterhoff here for another month cast, and I'm excited to have on as a guest today Adam Devander. He's the principal consultant at Every Developer and the author of Developer Marketing Does Not Exist, my favorite book on that subject. So, you know, totally jazzed to have you on the the show here and ⁓ to talk a little bit about where where we're at in developer marketing these days. speaker-1: Awesome. Thanks so much, Kate. Yeah. speaker-0: I want to begin with simultaneously the most difficult question I've been asked recently and ⁓ potentially the easiest. So and and I've had this, I had this pulled on me just recently. So, Adam, what are you telling folks who ask you to define what a developer is these days? speaker-1: Yeah, wow. ⁓ I I mean, I ha I have been saying that that definition is expanding and I mean it's been ⁓ at least more at least more than a year that I've been saying that. So I guess if it's continuing to expand, ⁓ I'm not sure it's hit everybody yet, but but ⁓ but quite a few are. I think there are still distinctions we can make to kind of ⁓ write code or at least ship code for a job versus ⁓ those who are maybe doing it as a hobby or to ⁓ solve ⁓ an initial problem, right? ⁓ I think there are distinctions there to be to be made still. ⁓ but maybe as that expands, that distinction will become less important. But ⁓ but I think you have still your kind of traditional used to write the code and now write the code alongside an agent, likely ⁓ developer and ⁓ and then others. And I'm happy to talk about either of those. I think the approach that a that a company that has a product for one of those, I think that approach changes a little bit depending on on who that audience is, for sure. speaker-0: Right, right, right. Yeah. So just in terms of the workplace, whereas, you know, a company that does use a lot of technologies, maybe not necessarily like a SaaS company or or exclusively like a software development company, but like it you know, at in the workplace, who now is self identifying as a developer in your experience? speaker-1: Yeah, and that's it's funny, that's actually a l slightly different question than the first one, right? That self identity because ⁓ because I do think that a lot of a lot of the folks who maybe haven't traditionally shipped code wouldn't give themselves that label because they ⁓ still feel the ⁓ the responsibility. I was gonna say baggage, it could be that, but the the responsibility that comes alongside speaker-0: Yeah, we're gonna build on this. Yeah. speaker-1: that label, right? Of of like knowing the software development lifecycle and feeling confident that they've ⁓ found edge cases. All of the things that that someone who is just looking to solve a problem might not feel as confident about, right? Yeah. ⁓ so yeah, and I and I've started to see companies center on this sort of builder label that is I think most of them mean It to be kind of a the expanded definition, right? That developers still have a place inside there. in some cases, some might truly mean this this person who is now doing stuff with code that does not have that, has not had that as a job before. So so I'm not sure everyone is using it the same way to mean this sort of expansion that includes traditional software engineers. ⁓ but ⁓ but in some cases they they are. And it's sort of okay, let's call it this so that then those that are developers can find our maybe more technical avenues toward this, ⁓ jump into the docs themselves, and then those who are more on the builder side will will use the tools that's ⁓ will kind of do it for them and and help them along. speaker-0: Okay. Okay. Well this is gonna help me to segue into trying to understand what it is that every developer does. So the definition of developer is expanding here. It seems like we might even get to everybody. Everybody is a developer. I don't know. T talk to me about what every developer is and does and and ⁓ how it fits into this conversation. speaker-1: Yeah. So we work with dev tool companies to help them figure out why developers traditionally would choose their products and what problems they're looking to solve with it, ⁓ which is kind of that first step that you need to understand to be able to ⁓ to have them choose your product. ⁓ the name itself has it was never meant to mean everybody as a developer or or that you want to reach every developer. it ⁓ it came out of so my background is I was the editor of Programmable Web, the first directory of APIs, watched that sort of if you've ever seen that hockey stick chart that used to be in every conference presentation. ⁓ you know, I and my team found those APIs to be able to generate that ⁓ that chart. And so one of the things that ⁓ that made that difficult was We were always in search of these new APIs, but it felt like there's some amount that really every developer needs to know. And so that's so when I left there, I thought, you know, what if I just zeroed in on those and we identified the things like great developer experience that they have in common and and could start to see like what are what are the things that people would would look at and choose. And so that's where so The APIs every developer needs to know was sort of the, ⁓ like let's let's have this at everydeveloper.com. So that's where the name comes from originally. Obviously what I do is different than that, but also very related in that we still care about what are the things that that those devs want from a developer experience standpoint. And ⁓ yeah, and what are the best ⁓ the best examples of that doing? speaker-0: Yeah. Well, I you know, I feel like you've alighted on a very good title for a company, ⁓ for the sake that it is kind of like ambiguously vague in a way that like allows for lots of potential meetings here. So this is good. It feels inclusive. I like, you know, I I I think it works. ⁓ okay, very good. All right. So we're setting up the question of like what is a developer? To put some of my cards on the table here. I've talked to folks like Sean Wang. about what it means to be an AI engineer and the fact that he's now including like the agents themselves as developers. More folks are doing that. I've written a little bit about ⁓ the these agentic teammates or AI ⁓ coworkers and and all this sort of terminology around that. And so I think that we've we're at this interesting nexus of trying to figure out what the heck a developer is, ⁓ what sort of identity is still gonna be authentic to this label as everything else is getting smooshy and and you know we've got PMs that are shipping things now and ⁓ you know it's just i it's it's it's moving quickly and ⁓ you know so we're trying to keep up but I I feel like you're you're very much at the forefront of this. ⁓ so I guess to to ⁓ yeah, ⁓ well let me let me first get your reaction to that. Can ⁓ you know, a computer be a developer? speaker-1: No. Yeah. I I can't say I've personally got gone that far yet because I still feel like they someone someone has to has to tell them what's important, right? And ⁓ and it feels like if you if you walk that back and you know, maybe it's the agents, agents, agents, and then finally you finally you get to that human who is who is there who is who's saying this is the direction we want to go. speaker-0: That's the rule. speaker-1: That to me is still the still the developer, but at some at some point it's it's semantics, right? It's like that agent does need to find things like a developer would need to find things, and they're going to need things perhaps slightly differently than ⁓ than that human would. ⁓ and I think some some might say drastically differently. I think that's another area we could we could dive into. speaker-0: ⁓ yeah. ⁓ yeah. okay. Well yeah, as a ⁓ you know, but let's let's do it. ⁓ I mean my f my first sort of th thought on this is like how do you market to this new breed of developers? Would that be a fair place for you to you to sort of share some of this wisdom you've been accruing? speaker-1: We're w are we talking about the humans here? ⁓ well and the speaker-0: I I'm thinking of like if developer marketing does not exist, who who what's the developer anymore? Sh do you need to rewrite this book? ⁓ where does that sit? So but no, I I I as we sort of hone in on this and and talk a little bit about some of the projects you're working on now, yeah, I'd be I would be happy to to take it both either direction. So what what is appealing to you? speaker-1: On on that title, which also s ⁓ sits with me the same way every developer does, right? I'm I'm stuck with it. I can't use that phrase developer marketing ever, right? Because I've already said it doesn't exist. But the ⁓ the idea there was ⁓ that's that developers don't want to be marketed to. They don't want to see that marketing, they don't want like they they want to be ⁓ educated and inspired, right? And I've had many people say to me, you know, by the way, ⁓ nobody likes marketing. Nobody wants to be mar you know, like ever everyone could be that ⁓ we might ⁓ yeah, we might we might be arriving at that same destination where it is it is the ⁓ you know, maybe it is the core the core piece of marketing to this person who wants to build something ⁓ with all of the many tools we have now ⁓ that yeah that they they don't want to be marketed to. They want to find the right tool at the right time and ⁓ know feel confident that it can help them and ⁓ and so ⁓ I think there are I mean there are a lot of tools now that are that are ⁓ helping them to be able to do that. A lot of times it is asking the agents That's ⁓ you know, where a lot of my attention has been recently is on LLM rank, ⁓ which is looking at when these developers ask these questions, what are the products that come back? And ⁓ and how does that how's that different based on the model that they're ⁓ that they're asking too? And so I mean, I've been for most of this year asking kind of the top three, so Gemini. OpenAI and Claude and ⁓ asking them about these popular developer categories and watching as it ⁓ changes and shifts. ⁓ and then the the real fun stuff is seeing how does that change as ⁓ the way you ask changes and the the way you ask. I mean, I think that's maybe one of the big differences between if we just take very broadly the traditional developer and the builder. that we're saying now, the new developer, the way they ask questions is likely going to be different also, or the things that they want. And ⁓ yeah, and and we're in a in a time now where a little tweak of language ⁓ can drastically change what you get back. speaker-0: This is stressing me out. I ⁓ yes, absolutely. You feel like a like talk about a wizard, right? You're like, I've got to get the incantation right or else it's just not gonna work. ⁓ okay, but to take a step back. So your the project you've been working on, LLM rank, ⁓ I will provide a link in the show notes. ⁓ but it's lm rank dot f yi, correct? Yep. Okay. And so are you considering this like a speaker-1: FYI, yeah. speaker-0: Benchmark? Like what do you what do you call like what is what is the nature of this project? Is this like sponsored? Like what yeah, what what is this like speaker-1: Sponsored, yeah, yeah. So it's it's research and it's actually a lot like the ⁓ to be full circle back to to ⁓ the starting of every developer of looking at what are these tools that ⁓ that people want to use. And ⁓ yeah, and it's I I mean I I share it with folks and I get back, ⁓ you know, really we should be number one, obviously, is the is the first thing people say. Like their product should be number one, or, you know, I think this this and that is ⁓ is wrong and like it might be, but I kinda have to say, like this is like there's some editorial judgment in there, some, certainly in how the question is asked, but really I'm trying to keep it at what are these models saying and then what can we learn from from their responses? And ⁓ then how can how can you use that to take that that next step and ⁓ Yeah, and so it's ⁓ it's yeah, fully I'm I'm running them and I'm sharing this data and I'm I'm ⁓ sharing what I learn as I go too. speaker-0: Yeah. Well, I have had a very good time checking out this site here. We love a little data, Red Monk, although ⁓ skeptical, right? ⁓ metrics with an MEH, you know. ⁓ we want you know, but I think what is very cool about this is we're living in this world of like new ⁓ benchmark. So I'm thinking of like Simon Willison's ⁓ he's got the pelican riding the bicycle thing. I maybe you're familiar. ⁓ and that's like his way of try trying out a model. And it's just like a silly goofy way of of like testing the model. ⁓ I feel like we had yeah w the the analyst at Redmunk should come up with with our own little goofy trial here. But ⁓ but yeah for for like a concrete example here. I'm looking at LLM rank and you've got ⁓ CDN. So if developers ask AI, which CDN should I use, ⁓ it always ranks Cloudflare at the top and then we've got Amazon CloudFront and Akamai and then Google Cloud C DN and then it the list goes on. But what's interesting is it's got like ⁓ you know the rankings go up and down. And then you you you break down which models ranked how. And so that's very fun. And you and so ⁓ I'm sure folks will want to dig into the methods. I don't know that we need to go through everything, but basically you're asking the model ten times about this and you don't share that your exact question because you don't want folks to game it. Am I leaving out anything important for folks to know to to kind of get it? speaker-1: No, that's it. Yeah. And and ⁓ asking the newest model. So ⁓ so making sure so there've been so the most recent numbers, July, is the first time that we haven't seen one of those models change. So so the the changes that you're seeing there are either ⁓ just the natural variability that you would see or like it seems like it's enough that there might be some Some subtle sort of tuning that is happening behind the scenes. I mean, there's not enough for us to really know whether that's the case. But they were fairly consistent ⁓ between those months. ⁓ but ⁓ but yeah, ⁓ being ⁓ asking those questions and then the the sort of next layer is ⁓ is that kind of what are the differences, what are the differences between specific use cases and things like that, which are not quite out there yet. These are numbers. I mean, this is what we've been doing with clients is looking at what are the different use cases and how does that change. and then that's to me the excitement. Yes, it's fun to see where the ranks change, but the real excitement is like when you ask for the open source tool and the entire top 10 just drops out and it becomes you know, the bottom of the ⁓ of the the ranks might move up because they were open source and the what were looked at as the category leaders are not. And then you have actual open source tools that are now part of the conversation that they weren't before when it was just this broader sort of ⁓ sort of topic. And open source is the one that I've looked at the most on there and ⁓ and yeah it's I mean in in a huge number of cases, ⁓ I'm looking here, yeah, the I mean in the so in identity management as an example, ⁓ you have that situation where the top ten disappears and five from the lower come up and then five net new products that are ⁓ that are a lot of those going to be like true like link to GitHub repo open source tools that might not have have been the thing that that the LLMs had decided the developer wanted when they were asking for a way to solve this because the quickest way sometimes is, hey, here's this service that off the shelf does it. But if you're asking for open source, then there's a different set of things that you want to achieve here. speaker-0: Huh. ⁓ this is fascinating. Okay. And so you're working with companies to help their ranking in terms of this sort of AI assisted discovery. speaker-1: So th yeah, that's on on the every developer side. So I mean we're not directly saying, Hey, work with us and we will change where things are on L L ⁓ rank. speaker-0: Personally are doing a little bit of that. That's a little bit of the consulting you're doing. speaker-1: Yeah, looking at what if you look at the delta between these, then you can learn, I mean, maybe you're okay that you don't show up on the open source because your product is not open source. And so ⁓ you're okay being essentially invisible to the developer who's looking for that. Uh-huh. But what we help help ⁓ DevTools with is understanding what where do they want to be visible. What do they think their positioning is? And do the LLMs agree is essentially the essentially the question there. And so yeah, when someone comes and says, we think we should be number one, well, right, I mean, right now the LLMs don't don't agree. Let's figure out what that means and what does it even should you be attempting to be number one in this category when there are all of these contextual ways to look at it? Where you might already be number one. speaker-0: Yeah, right, right, right. I mean, this is so fascinating to me as someone who admires your book, ⁓ which came out in what, twenty twenty-one, is that right? Okay. ⁓ because in that you talk about documentation and the importance of documentation to developer experience. ⁓ and now and and and sort of all the writing that goes around that. Blog posts, you know, social media, ⁓ having ⁓ speaker-1: Right, yeah. speaker-0: You know, the ⁓ all all the the sort of developer watering holes where you would expect developers to be looking for these answers, ⁓ which have been completely upended, of course. You know, Stack Overflow now is, you know, ⁓ it's it's it's not being used in the way that it has. ⁓ I don't want to say it's dead or anything, but ⁓ so what's interesting to me then is what you how how things have shifted. Is documentation still now the avenue to help folks who want to improve? how these ⁓ models are finding these dev tool companies. speaker-1: Right. Yeah, and I I I mean I don't wanna ⁓ I don't wanna go on a ⁓ on a definition tangents. Sure. But but you know, I think so thinking about documentation, developer experience and what is that if you just think about sort of the how it looks, like ⁓ a developer wants it to be dark mode and have the syntax highlighting and those sorts of things, right? Then ⁓ that stuff is not at all important to an agent who can't see, right? but I think I think it goes much deeper than ⁓ just the how it looks to like what matters. And so if you go back to the I mean the book ⁓ starts out with a developer experience chapter, like to set that foundation. Like this is where it starts. And of course written at a time that where agents were not the way that we were developing. But yeah, if you go back to those thirteen criteria that I lay out in the book, I mean, almost all of them is pretty directly translatable to things that an agent would need. And I'll so I'll give you like like three big ones off the top. Like, so accurate, up to date reference documentation. Right? Like I mean that was something that a dev needs and I think you'd be in a lot of trouble if you don't have that for your for your agents in in some way, right? They they ha need to have at least trained on that somewhere. Some accurate up-to-date ⁓ documentation must live. And ⁓ just based on the training ⁓ time frames, you would want to make sure that it isn't just counting on on that sort of training data. But then so in addition to that, you've got the the the actual context of the developer, whether that's whether that's the agent or the or the human developer. And ⁓ and that's like are there samples or something that that f frames the a use case? Like like is there some way to show this agent that the thing they want to get done can get done with this product. So some some kind of ⁓ framing around use cases like in in the ⁓ in the olden DX, we would say sample apps, right? ⁓ and then the other important piece of context is what's the language or stack? And like that a lot of times now is coming, I mean, the agent has that context in the code base that they're building in. And that is going having something that ⁓ is framed around whatever that stack is or the particular language is still important. And ⁓ I think having a having an SDK can still help that agent be able to integrate quicker than ⁓ than not. And ⁓ and that is still I mean that's straight out of The I mean the example I've given for years for framing around programming languages is to look at the Heroku Dev Center and you see those beautiful logos of programming languages. And if you are one of those if one of those languages is the stack you use, you're going to see that. You're going to say, This is for me. You're going to click on it and you're going to get the things you need, the documentation framed around that that language. And even though an agent is multilingual, doesn't doesn't have a a specific one that's their favorite, the code base they're in is going to have selected that at some point. And so you still want to be able to have that context. So those those are three of the thirteen and I think they're they're ⁓ still very valuable and you could argue more valuable than than they were before. I mean that first the updated and and accurate. I I mean, I don't know. Maybe people listening to this will say that's obvious, but ⁓ I think anyone who has ⁓ has integrated with anything will say y yeah, maybe obvious, but also really dang important. speaker-0: Yeah, this is super helpful. ⁓ okay. So the book still relevant. We're the the user potentially has changed. ⁓ so i if when more like when when these ⁓ dev tool founders come to you and they say, Hey, Chat GPT is recommending our competitor over us, what's the first thing that you're telling them to check then? How are they prioritizing this list of things that are are so important now that the agents are doing so much of this ⁓ recommending? speaker-1: Right, yeah. ⁓ and a lot of the talk now also is can that happen without the human at all? I mean, even more important, right, at that point. If if these agent loops are all making these decisions, then we don't even get a chance to say, ⁓ we wanted you to look at something else. ⁓ yes, ⁓ so for sure. So so coming with that with that ⁓ single model question, I mean that says and I think this is this is the I mean it's the classic developer way to be, right? To you sort of have your frame of reference and you stick in it. And so ⁓ so those founders might not realize that while they aren't being recommended by Chat GPT, they might be the number one choice of ⁓ Claude or Gemini or another, right? Or both or all, right? And ⁓ and that's I mean one of the things that's ⁓ that I've seen is just the so it's about half of the time in the LLM rank, so the 33 categories that that I've looked at, about half of the time all three models agree on a top three at least. Maybe not the order, but the the top three they agree, which sounds good, but that means that when someone is looking for your tool, DevTool Founder, that there's a 50% chance that if they asked a second model, they'd get a different recommendation. Right? That ⁓ yeah. Yeah. So ⁓ so being able to look, so I think it's it's a different message to the devs and and to the dev founders. To the devs, like adding a little bit of of robust search in here to, you know, right? Like speaker-0: That's big. speaker-1: Maybe there's a skill. So to put it in agent terms, there's a skill to choosing one of these tools. And it should be not just taking the first recommendation. It should be not just looking for a single model, the answer from a single model. ⁓ but looking for some consensus, making some comparison, ⁓ certainly give it as much context of what you want to achieve as possible. ⁓ And then yeah, to the dev tool founder, it's yeah, look look at that, but also make sure you understand where else devs are ⁓ are looking. And ⁓ and interestingly, Claude is often the outlier of of the three. It's not you know more than the other two are. and in my experience and talking to devs. Claude would also be the number one choice of model. So ⁓ so there's a good chance as a dev you are asking something that might disagree with another. And ⁓ yeah, and as a as a dev tool, if you're asking, you ⁓ you should get the full picture. ⁓ and then one other bit to the dev tool founder is to we often we talked about how the language can make a big difference. Like we often are asking the question we wish someone would ask. And so we are asking it in a way that might that maybe you would copy paste from your H1 on your on your homepage and ⁓ giving giving a little bit more thought into how that's asked. And some of that could be having multiple ways of phrasing it. I I heard a story from someone going to a ⁓ going to a company retreat and they they all wondered ⁓ where do we where do we sit when we when people when devs are asking about this so they all open their laptop they all go to their own favorite model they all ask their own separate question and then are surprised when everybody in in the circle there has essentially a different answer and that's That's because it's going to matter how you how you ⁓ how you ask that, right? And which model you ask. And so trying to trying to make sure that you take stock of which models are the ones that your audience is most likely to be using and ⁓ try to expand that beyond beyond one and then ⁓ really think about what are the questions devs would actually be asking for. So a lot of times that's that's more the problem and less the solution. It's the, you know, the classic ⁓ dev marketer problem is kind of leading with that solution with that feature and not the thing that's in the dev's mind before they've discovered you, right? Before they know that you you are the feature f flag solution. Before they know that you're the feature flag solution, you want to be saying, How can you experiment? How can I make a change with some of my users? That kind of approach. And ⁓ and yeah, and so bringing that into the same way you'd bring that into your messaging, you want to bring that into how you're asking these LLMs as well. Otherwise, some of the data you get is going to be. fully biased, probably toward you. Maybe it will make you feel better, but ⁓ it won't be what devs are really experiencing. speaker-0: And so if a dev tool company discovers that they are not ranking very high, you've got your 13 things that you suggested ⁓ for marketers to do historically. ⁓ is it just trying to like write a blog post that uses that sort of phrasing, create a little content, a YouTube video that then does satisfy that? Because the the models take a while to refresh, right? As opposed to like, you know, Google search or something. ⁓ I i it feels like maybe the payoff isn't as Quick the turnaround. speaker-1: And it's funny, SEO also is a long game, or historically was a long game. From what I've seen, it's a longer game, ⁓ this ⁓ this ⁓ the LLM ranking for sure. not only the the how infrequently ⁓ the training is they're retrains, ⁓ so the training cutoffs, ⁓ but also it seems like incumbents stick around, like they they stick around longer than and this is this is just sort of a like developer knowledge gut check of like so I I can't say for sure, right? But it like as an example, Bootstrap was still kind of in in the running. I mean it still is, but I mean it was until very recently was kind of like up top. ⁓ and even higher than tailwind, which in in my developer circles that feels like, ⁓ that's like two years ago. Right. I think that a lot of it is is just there's so much that they've trained on bootstrap for so long. People have talked about it. So that's which is going to be the case with every incumbent, right? That if it's already the or if it has been the de facto number one, then there's just gonna be more to latch on to. And ⁓ w one other piece could be that the models have not weighted newness higher in a way that that Google, I mean, this is maybe five years ago, ten years ago, right, said like this is important that you know you should go and update your your post and don't just put a new date on it, like actually change it ⁓ to to be able to still be relevant. I'm sure there's some of that weightedness. in in the L LMs, but it seems like it's not heavy enough for what we all would expect. And so incumbents are sticking around which which ⁓ yeah just means means longer longer timelines. I think that is still where the sort of thinking about it in terms of context versus category. You're not owning owning a f don't you know the the category leader will take a long time, but the context leader could be much sooner, as soon as the next training data. ⁓ also most of these ⁓ most of these LLMs will pull in recent search results. Like they will factor that into some of the results. Intentionally, LLM rank does not use that right now because it feels like that could ⁓ could kind of taint what we're seeing with the the actual training data. And the reality is the training data is what a model's going to fall back on if some of the the anything new isn't the newer context is not able to help. And so it's sort of like, how do we see this sort of foundationally? ⁓ but ⁓ but if you look at at sort of the the contexts, which would be things like use cases, then you start to see, okay, we're already we're at least performing better in the in this one or If we think that we should be the top Python tool for X, then and we're we're number three and we know that we have ⁓ better tooling than some of the others, then it's taking stock of, okay, are there places that we haven't said this? ⁓ we don't actually have a Python landing page where someone where a dev or an agent can go and find all of the resources that they would want from a Python stand ⁓ standpoint. ⁓ maybe that's maybe that's a change to make. Yes, you're it you'll you'll have to wait potentially to see that ⁓ come out across, you know, rankings, but I think you can see, and and we've definitely seen at the very least with a developer who has discovered you, if you provide them a page like that, then they're going to have a much better chance of being able to use Python in this case with your product. And ⁓ yeah, so there are there are moments along the way where you don't have to wait that long time, but recognizing if you're if you're on LLM rank right now and looking and saying, we need to be higher, we need to be higher immediately, that's that's going to take a longer time. speaker-0: Yeah, I think many vibe coders have had the experience of saying, Hey, I want to try out this new tool. ⁓ I'm thinking of like bundlers or something. You know, you say, Hey, I want to try out R S Pack or VET or something, and it's like, no, no, we're using Webpack. And then it'll just even like silently switch it back to Webpack because there's just so much documentation written around that. And this is happening so often. ⁓ I've also heard the experience that ⁓ some are having where, you know, if they've had a major change ⁓ with that. ⁓ you know, with that product, the old documentation is showing up and giving bad advice then. ⁓ is have you found a way to like overhaul that? Like is is the answer to like delete it or or is I don't I don't know, to like make it clearer that this is outdated or like so somehow tell the robots ⁓ that you know we create more robots files. I don't know. Yeah. What what are you what are you telling folks who are getting bad information from the agents? speaker-1: So I think correcting correcting the record is definitely the first step, right? And ⁓ and including that wherever you can. I think one who who ⁓ I've seen having this issue is Stripe, who is usually way ahead of the game in all all things DX, right? And you would think, ⁓ they have solved this, they've got ⁓ They've got developers, you know, they've got the agents using their ⁓ the right version of their APIs. Not the case, though they're still making sure to to l have it be visible. And it's one one thing. They're the only one I saw in an LLMs.text ⁓ and I went through this goes back to end of last year, went through for 30 some, I think the there were at that time twenty of the thirty ⁓ that I looked at that had that file and so went through Stripe was the only one who kind of spoke directly to the LLM at at kind of a section that said, if you are an LLM agent, please use the latest version of an API, look at ⁓ deprecation headers. You know, it gave sort of a a like this is what to do. There's still questions about whether ⁓ whether the LLMs use that LLMs.txt, but that's one place to put it. And and absolutely, I think that's a great example to follow to have that. Then I think you're seeing a lot of dev tools who are kind of throwing a lot of different things. They're creating skills. They're ⁓ C P servers for sure. MC speaker-0: P absolutely. speaker-1: I think these are these are all around answering exactly that question, which is the stuff that comes after what we've talked about. The actual discovery, right? Like that's yes, it's important, but then can they actually be successful? Can the agents be successful when they've found your tool? And yeah, that's where there are are a lot of these tools that primarily are human developer. Here is a thing, please give this to your agent. ⁓ and the the sort of maybe not spoken but ⁓ but kind of ⁓ message underneath is your agents are making the wrong choices, choosing the wrong tools. Please give them this thing that will help them make the right choices. And so I think that's why we're seeing a lot of those. And so yeah, you should ⁓ make sure that documentation can be easily copy pastable. ⁓ I think especially for that ⁓ that builder audience who if their agent isn't able to do it without that, they maybe will give it that sort of copy paste. And then after that, they're probably moving on to another tool on the list. Do you know anyone who is not this one that you told me that I can't get to work, right? So ⁓ so anything you can do to reduce that friction. And so that could be something as simple as copy pastes in these docs. ⁓ but ⁓ but yeah, might be MCP server skills, things that's ⁓ that will be able to orient hopefully the the documentation you already have that humans can get as well. Think things like here's the everything you know to get started, here are sample apps so you can see how we build and what we what ⁓ calls we make. Yeah. So I mean that's whether that ⁓ whether that makes a difference, I think, you know, is still still to be determined, right? And whether that makes it into training the next time so that these issues don't happen. And we we also know that ⁓ there'll be changes in the future that are also going to be after this this cutoff. So the more that we can ⁓ we can provide these tools, yeah, the better. ⁓ but right now it's still the human has to it in many cases point the agents at this tool that then will help them be able to use it better. speaker-0: And and I'm glad that you articulated that distinction and ⁓ between like making sure that developers, builders are successful once they have chosen the tool, and then the discoverability, you know, the these are separate concerns. You know, there's some overlap, of course, but these are these are things that both need to be addressed by founders. ⁓ okay, so I want to play devil's advocate here. So one could argue that on the the discovery side, ⁓ That any enterprise is not going to use Claude or Chat GPT to actually discover products, that they've got their seasoned senior developers, their engineering leadership, whoever, who are gonna be choosing. And so that isn't this is just hobbyists. Like who cares, right? This is not important. And potentially maybe it'll never be c ⁓ important. May I could imagine someone arguing that. So how do you respond to that? What why do you think discoverability is important? for from a purchasing side. Like why wh why is this part of a GTM strategy? Like P L G. You know, wa t talk to me about why should I care? speaker-1: Yeah, so I I mean let's let's first of all just assume that maybe they maybe they are right about tooling in internally. That will be the case. ⁓ but we have seen a long line of developers who are working on that side project who happen to also have, you know, senior architect role at at large corporations. So There are plenty plenty of examples of that happening ⁓ over the years. ⁓ now whether that whether that continues, ⁓ I mean, I I think probably the trend is likely too. We can also look back at at bringing your own device into work. And I mean, fifteen years ago there it was mostly like, no, we're gonna lock everything down, and now there's a realization that at some level, ⁓ you know. devices are allowing us, at least even post COVID, right? Like we we learned, ⁓ right, we need to actually be able to to ⁓ to get our work done distr in a distributed way, right? So I think similar things we're we're seeing here. Now, if the question is should there be other models that you ⁓ that you look at and see how how those are discovered there? 'Cause I think even that that senior team is going to be using these tools. They're not going to be saying saying, All right, we're only going to go and do our own non-agentic research on on the tools that we use. They're going to be using them too. So yeah, it should there be other models, you know, maybe. ⁓ it's it's a lot to keep in my brain at least for three right now. But ⁓ but as others ⁓ arrive for sure that's ⁓ that's a possibility. ⁓ yeah ⁓ but then I think it gets to also that audience and so if someone says ⁓ these these independent builders they're not important that's we've been seeing that in ⁓ in developer circles since way before ai. And that is it's a it's a natural tendency for for Companies that started on what ⁓ what people will will call developer first or bottom-up to move toward this more enterprise sales, top-down, you know, maybe developer last, if we want to be real ⁓ mean about the label approach. and ⁓ Yeah, so that's a natural progression. And I think if you're saying that, then you kind of do understand where that audience is, but it means you are cutting off this rapidly growing definition of developer. And ⁓ then at the same time you don't have access to that senior person who is working on that side project who's going to then bring that tool, which could be the best tool for for the job, into into the workplace. So it does seem short-sighted, but I think at least understanding who that audience is, and you can't appeal to anyone who would possibly use any tool ever. So you do have to sort of pick where those areas are that you're focused. And ⁓ and so doing that ⁓ not sort of haphazardly and sort of waving off, but doing that intentionally, I think is is really the real answer there and knowing who that is you want to reach and then making sure that you're in the places where those developers or builders are ⁓ are working and and playing. speaker-0: So as we wind down this conversation, I am interested in your predictions for where this is all going. So as the foremost ⁓ dev marketing ⁓ expert here, do do you see developer marketing as as maybe I don't know, ⁓ we're we it's definitely gonna evolve, but do you see that it maybe is gonna decline, become less important? Or do you see that this is sure it'll reinvent itself, but in fact it will become even more important? Because, you know, now everyone's a developer or because hey, we we have to try harder. It's even SEO is even on the longer time side. All all the challenges that we've kind of addressed here. Where do you see it going? speaker-1: Yeah, so I you know, I think if you if you look at w at the things that I say are important in developer marketing does not exist, it's really pointing toward engaging developers. I mean, the classic kind of dev rel ⁓ tools, right? And so from what we've even seen just in the last few years, there was there was a little bit of, ⁓ DevRel, that's less important now. ⁓ oops, it's actually the most important thing and our top thing that we should hire for. Like that's the trend that we've noticed in this ⁓ these last couple of years. And so it seems like that ⁓ that trend would continue. Whether we call that role developer marketing, whether the that is the audience that you're even marketing toward. I mean, there are there are certainly other audiences that that you could pick in there whose problem you solve, ⁓ But if it is a core developer problem that you're solving, then yeah, I think that developer engagement is going to matter. And so yeah, what we call that, I don't know. but ⁓ but that that is important even in in this agentic world that we're that we're moving into. Because again, there's someone who's saying the things that are important who identifies those problems to solve, and then There might be a whole team of agents that goes and needs the information and ⁓ as we talked about, most of that list is still important today and maybe even more important to the agents ⁓ than it was to the devs. speaker-0: All right, Adam. Well you have convinced me. ⁓ so before we we sign off here, ⁓ for folks who want to hear more, where are you directing them? What are your socials and ⁓ what what sort of links can you share? speaker-1: Yeah, you you can find me at ⁓ at LinkedIn for sure if you wanna ⁓ chat with me. I but then I think probably the best place to go and start to noodle around is lm rank dot fyi. speaker-0: Fantastic. All right. Well again my guest today has bad Adam Devander. My name is Kate Holteroff. I'm an analyst at Red Monk. If you've enjoyed this conversation, please like, subscribe, and review the monk cast on your podcast platform of choice. If you're watching us on Red Monk's YouTube channel, please like, subscribe, and engage with us in the comments.