Latest / Semantic SEO Podcast / Best Technical SEO Software 2027 (James Dooley Interviews Daniel Foley Carter)
Transcript
- James Dooley: Technical SEO tools for 2027. Now we're in the AI era and a lot of people talking about AEO and GEO. Is it just SEO? What's needed? What technical stacks are needed, um, with regards to search engine optimisation in 2027? Today I'm joined with Daniel Foley Carter, probably say the actual number one, definitely in the UK, for technical SEO. So I'm interested to hear, obviously now there's a lot of people trying to vibe code their own technical tools. They're using Claude. They're using ChatGPT. But I want to hear from the technical SEO guru himself, what does your tech stack look like going into 2027? Daniel Foley Carter: Yeah. So I'll be bringing along some of the tools I've used for a long time. So I think things like Screaming Frog, um, I'll carry on using it. I know that they've got their CLI and their MCP now, so you can sort of execute things remotely. Um, uh, so Screaming Frog. Um, other tools I use periodically. So Ahrefs, less for the technical SEO but more for the link side of things. Um, obviously SEO Stack because we built in... We started to build in the technical aspects of that. And what we're doing more on the side of is things like rendering, uh, LLM analysis, so DOM analysis and things like that. Um, and then I sort of... I've... I use a scatter. So I've used a mix of Lumar, I've used JetOctopus, um, Pi Datametrics. Um, the tools that I tend to sort of vary a little bit really depending on the size of the site that I'm crawling and how complex it is. Um, in terms of, like, my own solutions, would I vibe code my own solutions? Possibly, possibly not. I mean, I've made a few add-ons for Chrome, uh, to, you know, to quickly strip things out like internal link counts and things on pages. But my, my core technical SEO tools will stay the same. I mean, I think Screaming Frog is good primarily because you get a majority of the data that you need in one set. Um, although I would say we're in 2026 heading towards 2027, Screaming Frog definitely needs to revise how they prioritise certain things within their crawls because a lot of the things that are flagged as issues almost aren't issues. James Dooley: Yeah. And then with regards to, obviously you mentioned there SEO Stack. Obviously you've been building it now for half a century. It's been a long time and you're building it out. Um, it integrates into, like, Google Analytics and Google Search Console. Yeah. If someone only wanted to choose the two tools and, let's say, it was SEO Stack and, um, Screaming Frog side by side, a lot of people that try to do technical audits, the main issue is the actual reporting side of things. So, how... How does SEO Stack come with the reporting? Because it's easy to run a Screaming Frog audit, but it just looks messy. How... How can someone present it to a client to say, "This is top priority. This is least priority. Work down that list and get it done as soon as possible." Daniel Foley Carter: So, it's... It's going to be hard to say this, but I, I've... I've been a, a manual guy because I was doing the audits, like, manually doing 500, 600, 700-slide audits. So I'd crawl a site. I'd get all of the data out of Screaming Frog or, or Lumar or whatever I was using. Um, then I would obviously do, like, the GSC analysis. It was because of these things that I started SEO Stack because whenever I go to do an audit, the first thing that I look at before I look at the website, the content, the links, the page indexing, is I want to understand where they get their traffic from. Um, I, I like to do a segment analysis. So, I want to know where the traffic's coming from, brand versus non-brand, subfolder performance, last three months, year on year, so I can do a compare, so I can see where things are slipping or disappearing. But, but, but it was getting this data and then obviously turning it into something that can go to a client. So for most of the years I've always done it manually. Google Slides, take snapshots. But I have been trying to use AI and, uh, some people have seen over the last couple of months I've done quite a lot of studies and, and tests where I gave AI the task of doing an audit. Um, so obviously over the course of the year we've had evolution of AI models. Like we had GPT-6 Astra, uh, which then become Luna, where it, it could run its own console, so it could run its own browser. And then effectively you could give it a Google Slides doc and ask it to build an audit, right? So this is, like, the first sliver of going from manual to automation where I can take the data from Screaming Frog MCP, GSC API or SEO Stack MCP and get that fed in on an automated basis. But the problem is, is that the data is, like... I, I ran audits and I tried automation and it's just... You, you can't send that out to clients. You, you can't. So I'll give you a couple of examples, right? So I need to give an audit to a client, right? So, I'm still doing the legacy stuff of crawling the site and then getting that data and then getting the GSC data. Um, we know that GSC is a pain in the ass to work with because you get a thousand-row limits and you can't save your filters and all this stuff. So, SEO Stack is where I go to get all the GSC data and it's unrestricted. But at the, at this point in time, you know, uh, when I was using Claude and ChatGPT and I was testing them both to do audits, the problem that they had was context. So they can only process or get so much data before they try and do an audit. And what I was finding was Claude and ChatGPT both, because they can only process so much data in [inaudible], they would go over GSC API, but they could only make, uh, 25,000 requests at a time. Um, so what would happen is, is that because the model couldn't get every bit of data from GSC, it would do the audit, but it would miss out on loads of data. So it would make inaccurate assumptions of brand versus non-brand. It would put wrong data in. When it came to technical SEO, it was even worse. So it didn't understand things like canonical relationships. You know, it would take the assumption that externally blocked or internally blocked JavaScript were disastrous and things like this needed doing, when in fact if you actually manually looked at it, they weren't these problems. So I've tried the automation side of things and whilst, no doubt, in a year from now, two years from now, it will be dramatically better. At the minute, to, to build a report for a client, um, what I'm doing now is I'm using AI to connect to SEO Stack because it has the full data set. Um, it's, you know, it's not sample data like GSC is. So I can automate some of that analysis to, like, get top-level overview. Speeds up putting that into a Google Slides doc, but still the rest of it is quite a manual process at the minute because there's not enough advancement in these models yet. James Dooley: And then what about, um, GA4? Is it pulling... Is SEO Stack pulling from GA4 as well? Daniel Foley Carter: Yeah. So, so I, I, I can explain it. So when, when we... When I was doing audits and I was using GSC all the time, it was just... It was driving me mad because, you know, I wanted to layer filters and I couldn't, and I wanted to see beyond the thousand rows and I couldn't, and I got fed up with sample data. So I was having to use Supermetrics, GSC, uh, Search Analytics for Sheets. I was having to export this data and it was just such a headache. And I thought, why can't there be a viable GSC alternative? But the thing is there are GSC alternatives out there, like SEO Gets, GSC Wizard, um, there's other tools that connect to GSC. But we're, like... I believe we are one of the only data warehouses in the world that is a standalone GSC data warehouse. So if, if we're pulling in all the data and we warehouse it, we don't have the limits that GA4 and GSC does. So what I can do now is with GSC and GA4 together is I can do hybrid analysis. So I'll give you an example of that. I get a client, they've been hit by, you know, uh, core update or a spam update. So, what I can do is I can get all the URLs that have shown a significant drop. Then what I can do is I can say, right, all of those URLs that experienced a drop, now go and pull the GA4 metrics. So I can get average session, I can get time on page, I can get engagement rate, I can get bounce, and then it starts to become a lot easier for me to look at data at scale to say, hey, look, this content doesn't look very helpful. It's got poor engagement. That's probably why it's dropped out. Um, and I do this thing called engagement modelling. So, uh, what that is, is humans read on average 250 words per minute. That's, like, a global average. Okay. So a thousand-word article should technically take four minutes if someone is going to fully engage with that. So if you've got a thousand-word article and you've got 20 seconds of engagement, then clearly something is not right there, right? So SEO Stack was built to take away the restrictions. Now I get it. You can go to Claude, Gemini, ChatGPT. You can hook up GSC. You can hook up GA4. But we've made it so that there's no limitations on that data. And because there's no limitations on that data, it means when you, when you audit that data, you're auditing the full data set and not part of it. James Dooley: And does, um, does SEO Stack have got... Has it got an MCP to connect into if someone wanted to do it with Claude or ChatGPT? Daniel Foley Carter: Yeah. So, what, what are you finding then from at present? Because allowing people to run... I'm, I'm technically [inaudible] and I say this to everyone. Like, I'm, I'm great at business. I'm, I'm good on the off-page, on the link building. I'm pretty good with the semantics and the content. But when it comes to development and, and technical SEO, it, it's definitely my weakness. But I'm seeing people on the same bar as where I'm at, which is low, that are going, building and, and, uh, they're moving a load of WordPress sites onto Astro built by Claude. And they'll look nice, obviously, as long as they're removing all the eyebrows and all the rest of it. Like, they, they, they look pretty good. They look better than what they was. Um, but what, what technical problems are you seeing? Like, for... I can't see them because I ain't very technical, but are you seeing a big set of problems happening within the SEO community technically-wise? Daniel Foley Carter: Um, so it was worse. It has got a lot better. So I'll give you an example, right? When people started vibe coding websites, it was when platforms like Lovable first come along. So Lovable was, like, really quick off the gate. Um, and when, when people started, you... You could go to Lovable and you could build a site just with a prompt. Loads of people started doing it, but they didn't realise that the technology that Lovable was deploying the sites in was, uh, with something called client-side rendering. What that means is, is that, um, the website's pages don't actually exist as pages that are crawlable. They're dynamically generated. So people were deploying these sites that they built in Lovable. So maybe they've gone from WordPress or, or Wix or Webflow, and they've gone over to Lovable and they've deployed their site. And then because it's client-side rendering, the, the website was actually built with something called an SPA, which is a single-page application. It's like a web app. Okay, so it was never designed to be a website, but it looks and functions like a website, but Google and search engines couldn't crawl them. Um, so I, you know, I saw businesses that had lost, like, 98% of their traffic. You know, got that bad. Then, then as ChatGPT and, and Claude got better at vibe coding sites, um, we then saw that the output tended to be, um, React, uh, Next.js-based, but it was server-side rendered. So you could vibe code a site and at least Google could crawl it. But there are still... There are still, um, SEO problems that come up regularly. So, uh, some of them include things like vibe-coded navigations that, you know, don't have... That aren't, um, uh, server-side rendered or they're dynamically generated. Or where people ask for features in a page. Those features are vibe coded in as a JavaScript button element where search engines obviously can't interact with it, so the content behind it is not visible. And I've seen other things as well. So, uh, things like stuff like LLM accessibility, so the use of the ARIA labels and things like that. But in reality, more and more vibe-coded sites now are getting better off the bat. The one area that I am seeing, uh, uh, a regular problem is that when people vibe code sites and they give the content, I've seen it so many times where AI doesn't obey the full prompt and, and rewrites or changes content and people don't check it. Um, so those are the most common ones. Generally, the only other downside if you're vibe coding a site from a technical SEO perspective is generally if you're vibe coding a site, you, you don't have a backend CMS. And even if you build a rudimentary one, you've got to make sure that, you know, if you're dealing with canonicals, hreflang, you've got to make sure that those are done. Um, one of the best platforms, uh, is Claude deploying to Vercel, um, which is, which is one of the best because it's easy to set up 301 or 308 redirects. It's easy to put robots.txt. Um, and generally all the sites that I built in it do well organically. James Dooley: And then what about from, from, like, a technical... Because, like, surely now, with regards to what it seems like for me, um, in the trenches, it seems to be so many people are wanting to move away from WordPress, um, with the, the, the exploits of the plugins and stuff like that. Um, there just... Lots of reasons why they're restricted on what can be done. But, like, is SEO Stack going to start triggering and, and trying to be set up to bridge the gap from when someone moves from WordPress to an Astro site built by Claude that's been vibe coded, that there's going to be some things in place where it should be like, you need to make certain that the day it goes live you go and run a Screaming Frog audit and you're checking certain things within? Because that's where I think SEO Stack can grow. I think SEO Stack needs to move with the times. And I think moving with the times is, even if you don't agree that vibe coding's great, people are just going to do it. And then you've then got to fix the problems that it's causing in some way, shape or form. Daniel Foley Carter: Yeah. Yeah, absolutely. I, I, I think the thing is, is with something like, uh, SEO Stack, the, the ability to monitor and, uh, track impact is easily done. So you can set up annotations to say, you know, we've moved the site, um, and SEO Stack can track that and it can report back on what it can see. So that is possible. But I think a part of it is going to be education. So I'm... I've just set up a recording studio. Obviously not here, um, a temporary place at the minute, but in, in the office I've set up a recording studio. So I want to do videos on, like, uh, migrations, um, how you do them and then how you can use SEO Stack to your advantage. I think, I think the biggest thing is, like, when people, uh, go and do these things, it really depends on the kind of site, what they're moving away from, how big it is, what niche it's in. But it also really comes down to the expertise of, are they just asking ChatGPT, how do I migrate my site, you know, or are they getting professional support before they do? I mean, I've seen... I've seen lots of migrations in this year. Uh, I've never seen so many people migrating away from WordPress, but, uh, I've also seen a lot of them go wrong. Um, just the most silly things don't get, you know, attention paid to them. Like, they change URL architecture and then end up with redirect chains. Or, you know, because they vibe coded something, they've inadvertently had something coded that is not indexable or crawlable. So I've seen lots of different things, but you'll be able to go into SEO Stack, tell it that you changed from WordPress to, you know, Claude and Vercel or whatever it is that you're using, and it will track. So it'll be able to track indexing, page performance, and then it will be able to come back and, and highlight if there's been a significant change or if pages have dropped from the index. So that, that is all possible now. James Dooley: Yeah, for sure. Anyone who's listening to this, make sure you check out SEO-Stack.io. Um, it pulls in all the data. There's no restrictions from Google Search Console. There's no restrictions from, um, Google Analytics that's being pulled in. Um, I've had Daniel Foley Carter on. I wanted to... I'm actually going to be testing this in depth now because we've moved quite a lot of sites over Vercel. We've obviously still got a load of the WordPress sites. Um, we're going to... Yeah, we're going to push hard on it, if I'm being honest with you, Daniel. Um, we're going to make certain we're checking to see what the, the problems are. One thing I would say that, and I know you might not like this, but some, some people that are moving over and vibe coding these sites, I think something that would be good on the reporting side, it would be when, when you see the errors, if somebody's going to go all in on using, let's say, ChatGPT, using Codex or using Claude, would be almost having it where it could have a prompt to fix it, that AI could fix that problem. And I know you might say it, it gets it wrong, but some of these people, they just want to be hands-off and they just want to... And somehow they need... Somehow the prompt that says it got built, I can't log into the backend of the dashboard of WordPress anymore. I now need, whether it's an AI harness or Claude or, um, ChatGPT, to fix it. I need the prompt that says this is how it needs to be fixed. And it could be certain caveats in place, like make certain you double-check this, that you don't do this and you don't do this. And I think having that within SEO Stack with, or within the technical audits that you're going to be running, could be a game changer. Daniel Foley Carter: Yeah. So the, the interesting thing is, is that at the minute there's a built-in AI assistant, and that AI assistant is connected to your GSC and your GA4 data. If you've connected, um, you, you can just go and talk to it. So you can say... You can say, "I migrated my site two weeks ago. Can you give me an overview of indexing and page performance? Can you see anything? Is there anything of concern?" And it will go and it will look at the data and it will, it will come back. But what we are in the middle of building at the minute... So we got the agentic MCP, uh, which is really fun because, um, I hooked... So I shared this on LinkedIn. If you Google "cyber security SEO", our site comes up first, cybercos.com. Um, so I built agents using Hermes. So I got, like, a $10-a-month VPS, installed Hermes on it, uh, hooked it up to SEO Stack via the MCP, and then I gave it access to Claude and Vercel. So it could interact with a website. So it could deploy content, it could build pages, it could edit pages. Then what it could do is it could backwards and forwards with SEO Stack's MCP, and what it would do is it would... It would make a change on the site. So it would, like, write a piece of content, or it'd change a homepage, or it would change the nav links. It would tell SEO Stack that it did that, so it would log as an annotation. It would wait a period of time, and then it would go back to SEO Stack and it would say, what's happened? Like, has there been ranking growth, ranking changes? And then what I taught it to do was recursive learning. So what it would then be able to do is it would be able to do an implementation, leave it, it would go off and do another implementation, and then it would go back to the first implementation and it would say, right, what has worked? What hasn't worked? So the MCP allows that and you can build, like, insane automations with it. So mine are in my Slack. So every day they connect to SEO Stack, they get my rankings and they come back and they report on ranking changes. They report on impression changes, position changes. Um, but we're... We're actually just... We're close to deploying agents in SEO Stack, which you can tell the agent what it is that you're doing. If you're building a new site, if you're migrating, changing domain, and you can get that agent to do the work for you. So it can, uh, you'll be able to hook it up to your site, you'll be able to get it to interact and do things. So that's going to be... That's going to be the next thing that I think, because I think you hit the nail on the head there. More and more people now just want to save time with automations. You know, they, they just want to get things done quickly at scale. Um, I think, I think that's why so many people now, you know, they, they make Claude or ChatGPT or Gemini, uh, the core of their, their workflow, and then they just use MCP connectors for Ahrefs and GSC and things like that to speed everything up. But without rambling on too much, I think people do need to be really careful with all of this as well because the amount of times that I tested GSC with Claude and I got hallucinations. I got told things that weren't true. I even had it audit pages. So I, I, I tested it on one site. It audited pages that had been removed from the website, but because the pages had GSC data, it said... It gave me feedback for pages that were HTTP 410. So, I was like... And I did a post on it. I was like, you imagine if you automated your audit and you sent that to a client and they go, "We don't have these pages on the website. Have you audited them?" Then you're going to look... Then you're going to look really silly. James Dooley: Yeah, for sure. Anyone who's listening to this, what technical SEO tools will you be using in 2027? Leave a comment in the comment section. Daniel, it's been an absolute pleasure and we hope you like the episode on technical SEO. Cheers.