Latest / Full Tech Ahead / Scale AI Content Safely

Transcript

Scale AI Content Safely · Full Tech Ahead

• 0:19
  SPEAKER_00: Hello and welcome to Full Tech Ahead.
• 0:22
  SPEAKER_00: I'm your host, Amanda Razzani, and I'm excited to be here today with Chris Yates.
• 0:27
  SPEAKER_00: He is the SVP Product Design and Engineering at Pantheon.
• 0:31
  SPEAKER_00: How are you doing today?
• 0:32
  SPEAKER_01: I'm doing great, Amanda.
• 0:33
  SPEAKER_01: Thanks for having me.
• 0:35
  SPEAKER_00: Happy to have you on the show.
• 0:36
  SPEAKER_00: Can you share a little bit about the services that Pantheon provides?
• 0:41
  SPEAKER_01: Yeah, so Pantheon is uh, as we say, where the web works.
• 0:45
  SPEAKER_01: So uh we are focused on enabling organizations to build uh and ship on the web at scale, uh, which means both, you know, I described it in three dimensions.
• 0:59
  SPEAKER_01: It means that we have customers that need to build and run business critical websites that receive sometimes you know tens of millions of page views per hour in really spiky ways.
• 1:09
  SPEAKER_01: So that's the hard the vertical scale.
• 1:12
  SPEAKER_01: Uh we have customers that scale horizontally, they don't have one website, they have often hundreds or thousands of websites and they have to figure out how to manage through that chaos.
• 1:22
  SPEAKER_01: And the third dimension is people uh and the people that form the web team, whether they're technologists, they're editors, they're authors, they're designers, all have to come and work together.
• 1:32
  SPEAKER_01: Uh and that is a uh an interesting uh kind of ball, uh, especially when you add in kind of our new coworkers that are are uh toiling alongside us in the form of uh LLMs and agents.
• 1:46
  SPEAKER_01: Uh, but we have uh you know customers that have hundreds or thousands of people involved in their portfolios of websites.
• 1:53
  SPEAKER_01: So Pantheon's service is really oriented around serving the whole web team uh regardless of that dimension of scale or across all three.
• 2:03
  SPEAKER_00: Awesome.
• 2:04
  SPEAKER_00: Well, we're gonna talk today about the implementation of AI tools everywhere and the trust factor that comes from that.
• 2:14
  SPEAKER_00: So my first question is just from your experience, where is there a bottleneck when it comes to trusting AI and all these tools?
• 2:24
  SPEAKER_00: And how is this affecting teams?
• 2:29
  SPEAKER_01: I think the bottlenecks are just evolving uh or emerging, especially for customers that are, you know, especially like our customers, our users, the people that create websites that run on Pantheon are people that build websites that matter to their organization, to their business.
• 2:45
  SPEAKER_01: And those customers often have rigorous approaches to security and compliance and quality.
• 2:51
  SPEAKER_01: What we've seen with the kind of the this leap that AI tools have made just in the last six months is like this incredible uh improvement of the speed at which we can create things.
• 3:04
  SPEAKER_01: We can write code faster, we can create imagery and videos uh and text faster.
• 3:10
  SPEAKER_01: But now we've kind of hit this point of like, well, now we have to go push this all through the processes that we put up for good reason to create governance around the things that we do, to ensure that we are uh keeping our our data and our our customers' data secured to ensure that we you know meet the quality standards that we expect, whether that's for you know the the pros that uh we ship as marketing copy or editorial copy or you know stories uh on a news website uh or the performance of the site that uh we we code uh or design together.
• 3:47
  SPEAKER_01: So that you know is is an interesting challenge, kind of moving the the velocity of creation into kind of the the missing pieces of governance in some worlds, where we have to figure out a way to really create the scaffolding that enables that that speed to kind of carry through to the actual production realization of uh all of the magic that we get from these new tools.
• 4:11
  SPEAKER_00: Yeah, absolutely.
• 4:12
  SPEAKER_00: AI is speeding up, but how do companies see what's actually happening behind the scenes with AI?
• 4:19
  SPEAKER_00: How do they track that?
• 4:22
  SPEAKER_01: It's it's been tough in the world of the web because the you know, the way that websites have been built for you know 30 years uh you know really is multifaceted.
• 4:33
  SPEAKER_01: You have uh one element that is very much technical and it is driven by the code that you create.
• 4:40
  SPEAKER_01: The languages have shifted and and morphed over the years, but you know, it's it's still code kind of creates the scaffolding for how the site is going to look, how the site is going to behave, how the site is going to perform.
• 4:51
  SPEAKER_01: But then you have uh you know content uh that hydrates that code effectively.
• 4:57
  SPEAKER_01: And that comes through, you know, in the old days it was uh you know, you have a content management system that then spits out content and code and combines it and serves it on an Apache web server.
• 5:07
  SPEAKER_01: Uh, I'm kind of dating myself at that point, at this point.
• 5:10
  SPEAKER_01: But now it's you know, then we can't went into uh full stack kind of monolithic CMSs, Drupal, WordPress on the open source side, site core, Adobe AM, and and and so on on the commercial side of things.
• 5:22
  SPEAKER_01: And now we've kind of come full circle back to these decoupled architectures with you know APIs driving things like Next.js or Astro.
• 5:30
  SPEAKER_01: The challenge across all of those things has been that there are processes for looking into the code.
• 5:35
  SPEAKER_01: Most organizations, mid-market enterprise, are probably using something like GitHub or GitLab or Bitbucket.
• 5:41
  SPEAKER_01: They've got checks that run whenever you create a pull request, whenever you make a commit.
• 5:46
  SPEAKER_01: A bunch of things are going to happen.
• 5:48
  SPEAKER_01: And most of the time you can catch, you know, uh, because those tools are getting better at doing static and dynamic analysis, where there are bugs, where there are, you know, things that, you know, secrets or keys or et cetera, things that shouldn't be in the code that might impact security or performance.
• 6:04
  SPEAKER_01: But really, there hasn't been a great scaffolding around how we do the same with content and the intersection of content and code.
• 6:11
  SPEAKER_01: Uh, so that's been a big challenge in in the industry as we've just increased the velocity of creation of all of those things.
• 6:18
  SPEAKER_01: So that's one of the things that I'm spending a lot of time on personally, but also across the Pantheon organization.
• 6:24
  SPEAKER_01: We're spending a lot of time building that kind of next generation of tools to go give a deep view into both the content and the code base so that we can understand all of the changes, whether they're human changes or they're AI-driven changes, uh, and get kind of the bird's eye view within a site or across many sites.
• 6:43
  SPEAKER_00: From your experience, where are you seeing the most negative impact or things breaking down from people relying too heavily on AI without that oversight?
• 6:54
  SPEAKER_01: I think there's a there's always been this tension, I think AI or no, uh, you know, historically between kind of the the the needs of of the technology side and the needs of the, I'll just call it the business side in general.
• 7:07
  SPEAKER_01: You could call it it might be marketing, it might be editorial, whatever that is in in the organization, around kind of like the control and the speed and flexibility.
• 7:16
  SPEAKER_01: And I think what is happening is there are more shortcuts, there are there's more shadow IT, there are more things that attach to the side of organizations, uh, and you know, enable some additional ease or speed or kind of offloading of toil from creation of of things with AI, but kind of come through the side door versus coming in the normal process by which content or code changes get into the system.
• 7:43
  SPEAKER_01: And one of the other big challenges of that is inherent with that is that understanding what the tools are doing becomes increasingly challenging.
• 7:53
  SPEAKER_01: So I can ask Claude or Claude Code or uh Codex to create me a new website, to create me a new page, and it might be beautiful, it might be mostly functional.
• 8:03
  SPEAKER_01: Um, but then when I need to go change it, if I don't have the exact domain expertise, it becomes really difficult to go say, well, I I actually wanted to do this or I want to make sure it's not doing that.
• 8:14
  SPEAKER_01: That becomes super difficult.
• 8:15
  SPEAKER_01: So we really need to create scaffolding uh around how that works, uh, so that when an AI tool uh generates something for us, it's not just vibing and taking its, you know, all its compendium of every amount of human knowledge and using it to shape a picture that we want, whether that picture is literally an image or it's you know a prototype of a web app or a website.
• 8:40
  SPEAKER_01: We need something that allows us to kind of be on this kind of common plane between you know how the a human user can understand and then go and interact and edit that content, whether it's you know actual text content, it's imagery, it's it's a combination of things that come into a web page, or it's the code that sits below it.
• 8:57
  SPEAKER_01: So I think that that scaffolding and the kind of the challenge of of things coming in the side uh in this as people adopt tools uh very rapidly in organizations, um, you know, because it this it's a very exciting time to be in technology, sometimes terrifying time, but um a lot of people are trying a lot of different things at once.
• 9:19
  SPEAKER_00: Yes.
• 9:20
  SPEAKER_00: Well, what advice would you have for companies if they want to install some guardrails or some systems in place, make sure they stay in control of the AI implementation?
• 9:32
  SPEAKER_01: Uh I mean, first of all, if if they're not already implementing kind of a code review pipeline that has built-in kind of safeguards and checks where uh you know anything that's introduced into the technical side, the code side of your application, whether it's a monolith or it's a decoupled or you know uh application, that you know, you're applying those basic checks, whether they're available directly through a tool like GitHub or you're you're plugging them into other code quality and vulnerability detection tools, etc.
• 10:05
  SPEAKER_01: Uh the other side is a bit more challenging today, but I think there are there are safeguards that you can start to put in place around uh how you think of um kind of your organizational governance.
• 10:16
  SPEAKER_01: So it's less of like the hard stops, but more of the soft stops in terms of how you think about standardization of tools.
• 10:23
  SPEAKER_01: So instead of picking you know four different prototyping tools, let's you know say we're we're we're going to use this one, you can build uh some scaffolding around how those uh the code or the content is actually created.
• 10:36
  SPEAKER_01: Um, we've been leveraging custom skills that we create, uh, things like uh we've created kind of Pantheon internal skills, we've created an engineering codex that really encompasses a lot of the uh the customs, a lot of the rules that we use internally that we can kind of seed our AI tooling, whether it's related to engineering work or it's related to marketing work or it's related to content creation and with kind of a rule set to start from.
• 11:03
  SPEAKER_01: Those are like soft guardrails.
• 11:05
  SPEAKER_01: Uh, and we're working on, in terms of our product development, on harder guardrails around how that the actual integration of code and content work together to build websites where you know we have a structure that enforces, you know, uh basically a set of components that you can use that can be assembled by humans or assembled by agents or uh generative LLMs in order to kind of stay on that on the same page, use this the the similar, the the this the common substrate uh for humans and machines to interact.
• 11:39
  SPEAKER_00: I think another thing people are concerned about is sort of there's a lot of the AI slop out there is the term I hear a lot of yes and and very generic dry AI content.
• 11:50
  SPEAKER_00: So keeping brand voice and the vision of the brand, what advice do you have when it comes to that?
• 11:59
  SPEAKER_01: That's one of the things that we've built into uh it even just for our internal use, we've actually built into our design system.
• 12:06
  SPEAKER_01: So we have a really robust design system that our team built to govern like how Pantheon looks ass and works, and it enables us to do things amazingly quickly.
• 12:17
  SPEAKER_01: We have designers that can spin up new prototypes uh in minutes using uh generative AI tooling.
• 12:24
  SPEAKER_01: But we we also built into that is some of those guardrails around the corporate communication standards, around the product language that we use.
• 12:33
  SPEAKER_01: So we've baked that into that tool set.
• 12:35
  SPEAKER_01: So in addition to saying I, you know, I can prompt uh Claude uh or another uh tool to create a a new prototype that has really tremendous fidelity and usability and matches the standards of other products that we use that we are creating and are in production at Pantheon.
• 12:54
  SPEAKER_01: We also ensure that like the language that we put into that, whether it's what a button says or how we describe an alert, you know, how how that actually reads, uh, you know, how it uh is understood.
• 13:05
  SPEAKER_01: So we can create greater uniformity.
• 13:07
  SPEAKER_01: And that really is a tremendous shortcut to just the productivity of being able to do these things um and get those, get these prototypes in the hands of our users uh to get rapid feedback and and tighten the cycle of development.
• 13:21
  SPEAKER_01: But it also applies to things on the you know, more of the go-to-market side of our business.
• 13:26
  SPEAKER_01: And I think that's also, you know, regardless of what type of business you are, I think that's an opportunity to codify some of those standards about you know how you talk about your organization, uh, what you say, what you don't say, create those guidelines.
• 13:41
  SPEAKER_01: The next step really is something that we're working on as part of our product vision to enable that to just plug into the content management uh tooling, to be able to have that as basically a you know an agent that might continuously work in the background or a network of agents driven by your own rules, so driven by what the organization defines as kind of their standard for how content is created or how images are used or how media is presented or how code is worked on, uh, and enable that kind of continuous improvement process to go on in the background.
• 14:18
  SPEAKER_01: But you know, within the you know, the other thing that I think is really important uh as organizations are considering how to deploy AI created essentially products or output into production is do you have a process by which you can look at this stuff first?
• 14:34
  SPEAKER_01: How do you put a human in the loop?
• 14:36
  SPEAKER_01: How do you ensure a quality process?
• 14:38
  SPEAKER_01: All of these these things, whether they're automated checks on your code or they're automated suggestions and reviews uh of your content, uh, are wonderful, but I there's really no substitute for saying, wow, that looks right.
• 14:51
  SPEAKER_01: I can see the before and after very clearly, which means you have to have a structure for seeing the before you know what what the the state, the future state will be.
• 15:00
  SPEAKER_01: So being able to have you know a replica of of production or maybe several replicas of production where you have different work streams going on.
• 15:08
  SPEAKER_01: So having a system that that you can kind of pre-flight all of that, uh, all of the changes that you're making, I think is really critical as you're you're introducing a greater pace of change into your organization with AI tools.
• 15:20
  SPEAKER_00: Yeah, so having that human in the process is still pretty critical.
• 15:24
  SPEAKER_01: I I think so.
• 15:25
  SPEAKER_01: And like there's a lot of we can actually use AI to improve that and call attention to the things that have changed.
• 15:31
  SPEAKER_01: We can recognize the drift.
• 15:33
  SPEAKER_01: These tools are wonderful for that, but you have to have a place for that to run too.
• 15:37
  SPEAKER_01: So you have to have, you know, something outside of production that you can look at to say, yep, that's right.
• 15:42
  SPEAKER_01: Uh, you know, what the AI has has highlighted has changed, or what the deterministic script has highlighted has changed, you know, shows me what I need to pay attention to.
• 15:51
  SPEAKER_01: And now I can say yes, you can't just do that in in production.
• 15:54
  SPEAKER_01: So you have to have this, uh, have a tool set uh that enables you to have multiple versions of of your of your site or even of your fleet.
• 16:02
  SPEAKER_00: Absolutely.
• 16:03
  SPEAKER_00: Well, if there was one key takeaway you could leave our audience today with, what would that be?
• 16:08
  SPEAKER_01: I think that we've gotten to the excitement stage uh of this where like there's really rapid adoption.
• 16:13
  SPEAKER_01: I think we've we've turned the corner in most organizations from I don't know about this to oh my gosh, I need to use this everywhere.
• 16:19
  SPEAKER_01: I think my my recommendation is don't wait on the next step, which is how do we you know drive governance and how do we put these guardrails on uh how we're we're doing things?
• 16:31
  SPEAKER_01: It doesn't mean necessarily slowing them down, pulling, you know, you know, putting on the brakes on the whole thing, but being thoughtful about uh you know what is the human loop process for evaluating uh these changes.
• 16:44
  SPEAKER_01: Uh, is there are there some fail-safes that we can introduce?
• 16:48
  SPEAKER_01: Are there some ways that we can generate efficiency?
• 16:50
  SPEAKER_01: Because now I uh in most organizations that and other with other technical technology leaders I've talked to, there's experimentation going on everywhere in every single department.
• 16:59
  SPEAKER_01: And I think you know, calling having those people come together and and also have some key leaders drive some standardization is super helpful because that will just drive efficiency and safety and governance across the organization.
• 17:12
  SPEAKER_00: Wonderful.
• 17:13
  SPEAKER_00: Well, thank you so much for coming on the show today and sharing your insights.
• 17:17
  SPEAKER_01: Absolutely.
• 17:17
  SPEAKER_01: This is great.
• 17:18
  SPEAKER_01: Thanks, Amanda.
• 17:19
  SPEAKER_00: And thank you to our audience.
• 17:20
  SPEAKER_00: If you have any questions or comments, of course, leave those and I'll try to reply back.
• 17:24
  SPEAKER_00: And until the next podcast, have a wonderful week.