Latest / The Adults In The AI Room / Lief Erickson Navigates the Complexities of Content Architecture with AI - The Earley AI Podcast with Seth Earley - Episode #045

Transcript

Lief Erickson Navigates the Complexities of Content Architecture with AI - The Earley AI Podcast with Seth Earley - Episode #045 · The Adults In The AI Room

• 0:00
  SPEAKER_00: This is the Early AI Podcast, hosted by Seth Early and Chris Featherstone.
• 0:05
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as CMS Wire.
• 0:24
  SPEAKER_00: Now, enjoy the show.
• 0:31
  SPEAKER_02: Okay, well, welcome to the Early AI Podcast.
• 0:35
  SPEAKER_02: My name is Seth Early, and I'm here with Liam Kunik, who is our guest co-host today, I would have to say.
• 0:41
  SPEAKER_02: Hello, Liam.
• 0:42
  SPEAKER_02: How's it going, everybody?
• 0:45
  SPEAKER_02: So Liam helps with the pre-production and post-production.
• 0:48
  SPEAKER_02: Chris Featherstone couldn't make it today, so I had Liam join us.
• 0:52
  SPEAKER_02: So today we're going to discuss content strategy and content architecture and information architecture as they relate to large language models and generative AI.
• 1:03
  SPEAKER_02: And our guest today has uh decades of experience in technical documentation and information architecture in DITA.
• 1:12
  SPEAKER_02: If you're not familiar with DITA, it's Darwin Information Typing Architecture.
• 1:16
  SPEAKER_02: And he has uh done a great deal of work in the across industries and working in content strategy as a master's doing content strategy.
• 1:25
  SPEAKER_02: Please welcome Leif Erickson.
• 1:28
  SPEAKER_02: Thank you for uh joining us today, Leif.
• 1:31
  SPEAKER_02: Seth, I am really looking forward to our conversation.
• 1:34
  SPEAKER_02: Wonderful.
• 1:35
  SPEAKER_02: Wonderful.
• 1:35
  SPEAKER_02: So I wanted to start off with what do you what would you say are some common misconceptions around generative AI or large language models and content specifically?
• 1:44
  SPEAKER_02: What are your what are your thoughts there?
• 1:46
  SPEAKER_01: Uh well, I think the first thought is that it's a a magic bullet that is going to solve all of the existing content problems that have existed in organizations up to this point.
• 2:00
  SPEAKER_01: It's not it's not a magic bullet.
• 2:06
  SPEAKER_02: Why do people why do people think that?
• 2:08
  SPEAKER_02: And and what problems, how are they why why are they thinking that?
• 2:12
  SPEAKER_02: And then what problems are is that going to lead to?
• 2:15
  SPEAKER_01: They're thinking that, I suppose, because uh some marketing folks are really good at their jobs.
• 2:21
  SPEAKER_01: Uh and and in specialized use cases, very constrained demos, uh the technology looks fantastic.
• 2:30
  SPEAKER_01: And it it can be, but when we start dealing with exceptionally large corpus, uh large content sets that uh the organizations that we tend to deal with, then uh the problem starts to be one of curation, of uh validation, and we end up with all sorts of errors or uh unclear responses coming out of the generative tech.
• 3:03
  SPEAKER_02: So it's interesting, you know, when you when you uh uh look at generative AI and you look at the problem of content and large volumes of content, it is a context problem, right?
• 3:15
  SPEAKER_02: Just like we've always had a context problem.
• 3:18
  SPEAKER_02: If you ask an ambiguous question, you're gonna get an ambiguous result.
• 3:22
  SPEAKER_02: And the same thing happens with large language models, which is why the whole idea of prompt development, prompt engineering is so important.
• 3:29
  SPEAKER_02: So tell me a little bit about what are your thoughts about um, so there's a misconception that's going to solve all your problems.
• 3:36
  SPEAKER_02: What else?
• 3:36
  SPEAKER_02: What else are you running into uh in the industry?
• 3:39
  SPEAKER_01: Well, the uh the other big issue that we run into is why do you want to do the problem or why do you want to do the project in the first place?
• 3:49
  SPEAKER_01: Like what's the business case for this AI project?
• 3:54
  SPEAKER_01: And uh we are seeing that executives are struggling to come up with uh the reason why.
• 4:03
  SPEAKER_01: You know, the one that is justified in the the the dollars and cents of it all.
• 4:10
  SPEAKER_01: And right now, I think what everyone is doing is chasing the the hype, uh wanting to not be left behind.
• 4:19
  SPEAKER_01: And I think if we zoom out, there's there's I'm really speaking to two different issues there.
• 4:27
  SPEAKER_01: Uh the the number of folks that are really dealing with a generative AI in a systematic approach that have the business case for it is very small, at least what we're seeing.
• 4:45
  SPEAKER_01: The number of folks that think it's a great idea worth pursuing and don't have that business case is a much greater number.
• 4:56
  SPEAKER_01: And that first group, that first cohort is they're going to be fine.
• 5:04
  SPEAKER_01: The second group, the much larger one, is really going to have problems and struggles as they go through the process.
• 5:14
  SPEAKER_01: I think they're first going to approach it as a technology project.
• 5:18
  SPEAKER_01: And they're going to get to a certain point and things are going to look great.
• 5:22
  SPEAKER_01: There's going to be lots of progress.
• 5:24
  SPEAKER_01: The engineers are going to be connecting things together, spitting things out, seeing, look, we can do something.
• 5:31
  SPEAKER_01: And then the rubber is going to hit the road, and there is going to be uh erroneous answers.
• 5:40
  SPEAKER_01: You had mentioned earlier, context uh and intent that is going to be uh open to interpretation, and the executives are going to wonder what they got themselves into.
• 5:58
  SPEAKER_02: So, so when you start thinking about um building that business case, I mean, what have you seen or what do you think the business case should look like?
• 6:08
  SPEAKER_02: And I and I totally agree with you.
• 6:10
  SPEAKER_02: I I agree that there's a lot of fear of missing out.
• 6:12
  SPEAKER_02: There's a lot of executives saying, well, we have to do something.
• 6:15
  SPEAKER_02: We have to do something with Gen AI.
• 6:16
  SPEAKER_02: Uh, we have to show that we're leading edge or have to show that we can, we're AI savvy.
• 6:21
  SPEAKER_02: Uh, but then when it comes down to it, it's like, okay, what do we really do?
• 6:24
  SPEAKER_02: What problems are we really trying to solve?
• 6:26
  SPEAKER_02: And as you mentioned, you know, there's a lot of technology firms or vendors that are offering solutions that are more marketing hype.
• 6:34
  SPEAKER_02: You know, they're communicating stuff saying, oh yeah, you know, we'll we'll put all your content in here and you'll have you know much greater access, much more precise access, et cetera.
• 6:44
  SPEAKER_02: So so the question is, what what do you see as a high value use case that people should be thinking about and making that business justification?
• 6:53
  SPEAKER_01: I well, I don't know uh every business and what their their needs are.
• 6:59
  SPEAKER_01: So the the executives themselves are going to have to know what their problems are, and we can help them define that and we can help put some parameters and uh some objectives and metrics around what those look like.
• 7:15
  SPEAKER_01: But are they looking to increase brand loyalty in some way?
• 7:22
  SPEAKER_01: Are they looking to be more efficient?
• 7:26
  SPEAKER_01: Uh are they looking to reduce support calls?
• 7:29
  SPEAKER_01: Do they need to manage risk?
• 7:31
  SPEAKER_01: We'll come back to risk almost certainly in our conversation.
• 7:34
  SPEAKER_01: Do they need to increase revenue?
• 7:38
  SPEAKER_01: What is it that they're looking to achieve with their AI project?
• 7:42
  SPEAKER_01: But that those issues that I just raised, or the the objectives, those apply for any project, not just an AI project.
• 7:53
  SPEAKER_01: So it's it's why are we doing this in the first place?
• 7:57
  SPEAKER_01: And what does success look like?
• 7:59
  SPEAKER_01: Right.
• 8:00
  SPEAKER_02: I think that's a great point.
• 8:01
  SPEAKER_02: And and you know, a lot of organizations are approaching this for support, customer support.
• 8:07
  SPEAKER_02: And it makes sense to do that.
• 8:09
  SPEAKER_02: But again, we need the context, and we're not, it's just not gonna work as expected if we just unleash it on all of the content.
• 8:16
  SPEAKER_02: You know, the example I use when I explain this is if you are looking to troubleshoot, you know, an error code and you have a certain device and the error code is 50.
• 8:25
  SPEAKER_02: Well, if you're on the phone with somebody, you know, you can't just say, what's error code 50 mean?
• 8:30
  SPEAKER_02: Right?
• 8:30
  SPEAKER_02: They that's an ambiguous question.
• 8:32
  SPEAKER_02: They're gonna say, well, what model, uh, what device do you have?
• 8:35
  SPEAKER_02: And what model and what's the configuration?
• 8:37
  SPEAKER_02: And the same thing if we're using a virtual assistant or we're using a chatbot, the chatbot doesn't have that context.
• 8:43
  SPEAKER_02: It's gonna need that context.
• 8:45
  SPEAKER_02: And on that piece of content, we have to tag that content with that context.
• 8:50
  SPEAKER_02: We have to say error code 50 for this type of modem or router or bridge for this type of device with these configuration parameters.
• 8:59
  SPEAKER_02: That all has to be there because otherwise there's no context.
• 9:02
  SPEAKER_02: So maybe we can talk a little bit about content architecture and things like data.
• 9:07
  SPEAKER_02: For people who are not familiar with data, um, let's talk about that, maybe uh define it and uh uh talk a little bit about how our content architecture helps.
• 9:17
  SPEAKER_01: Right.
• 9:17
  SPEAKER_01: So for folks who are not aware, DITA is uh an XML framework for identifying semantic chunks of information.
• 9:30
  SPEAKER_01: Uh so with the the XML, you would have semantic tags uh describing the content as a set of steps, maybe, so or uh reference material, or more specifically a field setting, for instance.
• 9:49
  SPEAKER_01: You could you it's not part of the the out-of-the-box ditta, but you could start to tag your content semantically that this is uh a field setting, or this is an error code, and everything within the error code elements or tags would be findable then by our machines that are going to uh extract the information out of the content set.
• 10:15
  SPEAKER_02: So when you think of content architecture, it's telling you the is-ness and the about-ness.
• 10:19
  SPEAKER_02: What is this thing?
• 10:20
  SPEAKER_02: This is a piece of documentation, and what is it about?
• 10:23
  SPEAKER_02: It's about this particular modem, it's about this particular router, it's about this particular error code, whatever it is, right?
• 10:30
  SPEAKER_02: Or troubleshooting step and so on.
• 10:32
  SPEAKER_02: But what we have found is that you know that uh that kind of tagging does improve the the recall and precision for the LLMs.
• 10:40
  SPEAKER_01: So Seth, can I jump in here and add a little bit more to that?
• 10:44
  SPEAKER_01: And it think of think of that uh aboutness or isms, the way uh you might think about uh if you were to need to find a a book in a library.
• 10:57
  SPEAKER_01: You walk into a a library, uh some people still do that, and the books are organized, they're organized following a particular system, and you just need to find where the book is located in the the the rows and the shelf.
• 11:19
  SPEAKER_01: The structure within the the markups help the machines and humans find that information.
• 11:30
  SPEAKER_01: So it helps both with the the input uh and the and the output uh of from a generative AI perspective.
• 11:38
  SPEAKER_01: It helps the machine understand what and where the content is located, and then uh when it combines that with uh other technologies to generate the the response, it helps then the the human on the other side of that query get their answer so they can go about what they were looking to achieve.
• 12:01
  SPEAKER_02: So this kind kind of comes to this whole idea of retrieval augmented generation.
• 12:06
  SPEAKER_02: Do you want to uh define that for people?
• 12:08
  SPEAKER_02: And you know, they're probably somewhat familiar with it, but I want to hear it in your words how how would people think about, because you just talked about the fact that you're finding something, you're using these tags to find something, you're trying to find the book in the library, what shelf, you know, what room, what shelf, what you know, uh, etc.
• 12:26
  SPEAKER_02: Uh so talk about what that means with regard to uh Chat GPT types of applications and large language models.
• 12:34
  SPEAKER_01: I think we need to sort of separate that into a couple of different aspects.
• 12:39
  SPEAKER_01: Uh and I think we need to start with the understanding that the response that ChatGPT or other systems is going to produce doesn't exist in the original source content.
• 12:55
  SPEAKER_01: So what ChatGPT is producing is made up.
• 13:01
  SPEAKER_01: That is the generative, regenerative part of the retrieval augmented generation.
• 13:09
  SPEAKER_01: So if we back up uh into what was the process, we don't say it's made up, it's made up based on its understanding.
• 13:18
  SPEAKER_01: It's not uh we're not saying necessarily that it's fictional, we just are saying that uh this uh uh response uh doesn't exist anywhere in its source content data set.
• 13:33
  SPEAKER_02: Yes, uh and and the answer is coming from the large language model's internal representation of concepts and terminology and relationships about the world, right?
• 13:45
  SPEAKER_02: Based on content it's been trained.
• 13:47
  SPEAKER_01: Right.
• 13:48
  SPEAKER_01: So uh let's then understand what let's break that down a little bit.
• 13:53
  SPEAKER_01: And this exchange started with a human putting a uh a query into uh the system, and that query has a number of words in it, and the chat GPT takes that and under, you know, starts to break it down into intent based on a number of other things that we don't have to get into here, but that then goes back to its source corpus and starts to retrieve information that is relevant based on the keywords and entities that it is finding in the the original query.
• 14:37
  SPEAKER_01: Behind the scenes, there's some additional information being added around the prompt that gives it some context, but uh but that's the process of the retrieval, the augmentation before you get to that generative made-up response.
• 14:54
  SPEAKER_02: Right.
• 14:55
  SPEAKER_02: So, so it's interesting.
• 14:57
  SPEAKER_02: You know, I I think of it as you are, as you said, you are defining the intent of that query, right?
• 15:04
  SPEAKER_02: You're trying to resolve it just like when we have a uh chat bot and we have utterances, they're different ways of expressing an idea or concept.
• 15:13
  SPEAKER_02: You know, my my password doesn't work, I'm locked out of my computer, I forgot my username, right?
• 15:18
  SPEAKER_02: Those all mean the same thing, right?
• 15:20
  SPEAKER_02: Change password, update password, reset password, whatever it is.
• 15:23
  SPEAKER_02: And then we're doing that on on steroids, right, with with a large language model, because some of these prompts can be quite complex, right?
• 15:31
  SPEAKER_02: And you can stack prompts.
• 15:33
  SPEAKER_02: And you know, I'm I'm doing a course right now where we're looking at building these really extensive prompts.
• 15:40
  SPEAKER_02: And what's interesting about those is that the LLM has to say, well, what does this really mean by putting those prompts into a vector space and then saying what's near this vector space in terms of other content, in terms of related content?
• 15:55
  SPEAKER_02: Now, if it does it based on its own understanding of content, that's one thing.
• 16:00
  SPEAKER_02: But as you say, that can generate erroneous results.
• 16:03
  SPEAKER_02: But instead, if we're retrieving some information from a trusted source, then we're actually taking that information and then making it more conversational with the LLM.
• 16:14
  SPEAKER_02: So it's a matter of saying that those tags and those uh and that architecture and that structure adds the ability uh to fine-tune what those results will be and to make sure that it's valid, right?
• 16:28
  SPEAKER_02: It's coming from a vetted quality information source within the organization.
• 16:33
  SPEAKER_02: So it's interesting when you think about the retrieval piece, you know, organizations compete on their knowledge, right?
• 16:39
  SPEAKER_02: So it's it's about it's not if you if everybody just used uh an LLM to answer questions, well, they're not getting any differentiation, right?
• 16:48
  SPEAKER_02: They differentiate on their knowledge.
• 16:50
  SPEAKER_02: That's competitive advantage.
• 16:51
  SPEAKER_02: So the idea is to say use your own knowledge, your own content, and structure that content in such a way that the LLM can make the best of it.
• 17:00
  SPEAKER_02: So that's kind of you know where this is extending to.
• 17:04
  SPEAKER_02: Do you have any more thoughts and context on that?
• 17:06
  SPEAKER_01: Yeah, you you made me think of a couple of different things.
• 17:10
  SPEAKER_01: Uh, you'd mentioned uh search, and you know, that is certainly one way that people are going to be using chatbots now is that you know, with uh and the the virtual assistants, it's that we've been trained over the last 25 years of internet searches getting progressively better.
• 17:34
  SPEAKER_01: For for those of us that have been around a while, we have seen how searches have improved.
• 17:42
  SPEAKER_01: Uh, I play uh a sport called squash, but squash in and of itself is somewhat of an ambiguous term because it has a meaning uh in a botanical sense that is very different from a sport sense.
• 17:58
  SPEAKER_01: And 25 years ago, putting search into uh you know Yahoo at the time or Google returned ambiguous results.
• 18:10
  SPEAKER_01: I would be getting recipes for uh uh squash and very little about the sport that might have something to do with the number of people who play the sport, but that's irrelevant.
• 18:24
  SPEAKER_01: But it's it's dealing with uh search and intent and uh accuracy.
• 18:31
  SPEAKER_01: And over the years, searches results have gotten better and better and better because search engines have gotten better at uh determining intent out of our queries.
• 18:44
  SPEAKER_01: You know, that did you mean uh instances that started coming up many years ago now?
• 18:51
  SPEAKER_01: The other thing that you made me think of is the idea that the quality of the data, and you've spoken to well, knowledge is what uh separates one organization from another organization, and it is that taking your organization's information and pairing it with a particular LLM is what is going to be the the technical connections of what will make your virtual assistant or your chat bot respond differently than uh somebody else's.
• 19:30
  SPEAKER_02: Yeah, and and and that's what it's all about again, because we're trying to have competitive differentiation and standardization gives us efficiency, but uh differentiation gives us competitive advantage.
• 19:42
  SPEAKER_02: Our competitive advantage is based on our content and our knowledge and our expertise, knowledge of customers, their needs, routes to market, the technical solutions, the competitive landscape, market, you know, ecosystem, and so on.
• 19:56
  SPEAKER_02: So let's see.
• 19:57
  SPEAKER_02: Um, when we start thinking about organizations that are looking at content strategy and being more serious about this, I mean, do you think that they you know what are you seeing is driving the need these days for content strategy and componentization versus what we'll see in the future, which I think is going to be more about retrieval augmented generation and having that as a reference point.
• 20:22
  SPEAKER_02: But what are your thoughts about what's driving today?
• 20:25
  SPEAKER_01: It it comes to efficiency in in many cases.
• 20:29
  SPEAKER_01: Um it's how if we're a multinational organization and we translate and localize our content, you know, having structured content makes uh translation and localization significantly easier, not necessarily easy, but significantly easier.
• 20:49
  SPEAKER_01: And uh that can lead to um massive savings over uh unstructured content, uh translation, localization.
• 21:00
  SPEAKER_01: Another reason is managing risk.
• 21:04
  SPEAKER_01: If you're in a regulated industry and you have content that has to be uh written in a very precise and particular way, having the a very finite set of or discrete set of information that needs to go through that review process to help manage and control that risk.
• 21:27
  SPEAKER_01: That that would be another reason.
• 21:30
  SPEAKER_01: Another thing is maybe you have content that can function in multiple places.
• 21:38
  SPEAKER_01: Uh, so the context is different, and that allows you to produce more products and therefore sell more.
• 21:48
  SPEAKER_01: So having and that increases revenue.
• 21:52
  SPEAKER_01: So we've just talked about you know three different ways that from a managing risk, uh reducing cost as well.
• 22:00
  SPEAKER_01: Is increasing revenue that is all supported through structured content.
• 22:05
  SPEAKER_01: And if we zoom back out, the strategy.
• 22:08
  SPEAKER_01: So the structured content is how we go about achieving the business objectives and goals and the strategic desires.
• 22:18
  SPEAKER_02: Talk a little bit more about that.
• 22:19
  SPEAKER_02: And uh Liam, I know you have a question, but talk a little bit uh more about that, Leaf, in terms of increasing revenue.
• 22:25
  SPEAKER_02: Where are you seeing the increased revenue play with uh structured content?
• 22:30
  SPEAKER_01: Well, it was just like I was I just spoke to uh if you have a component, uh oftentimes we talk about documentation and specifically in the the structured authoring world as little bits of Lego blocks, and that you can come you know combine them in different ways to produce different things.
• 22:53
  SPEAKER_01: Well, that idea isn't unique to structured authoring, that is also how many products, engineered products, software or physical products are built as well.
• 23:08
  SPEAKER_01: So what we are doing is with our structured content is really matching what's already happening on the engineering side.
• 23:17
  SPEAKER_01: It's just that that same level of thinking hasn't been brought to bear on the documentation sets in as many organizations as that thought process and thinking is with the engineering side.
• 23:33
  SPEAKER_01: And so, in some ways, our industry is catching up, our practices are catching up to other things that are already happening, and that all goes to velocity and scalability of an organization.
• 23:46
  SPEAKER_01: An organization always, most organizations always want to be growing and doing things faster on a shorter time frame.
• 23:54
  SPEAKER_01: And in order to do that, in order for the documentation team to meet those objectives, they structured content really helps move the needle forward for them in that in that way.
• 24:08
  SPEAKER_02: Yeah, and I and I suppose you know the way we look at some of the personalization approaches, they use structured content to experiment with different variations of messaging, right?
• 24:17
  SPEAKER_02: Different call to action, different hero image, different target, different uh value proposition, and so on.
• 24:22
  SPEAKER_02: And the more personalized we can make that content and the more appropriate the content is, uh, depending upon where someone is in their life cycle, where they are in their journey, is going to improve conversions, right?
• 24:36
  SPEAKER_02: Which will lead to increased revenue.
• 24:38
  SPEAKER_02: So I think that's a really important piece.
• 24:40
  SPEAKER_02: You know, there's a lot of things.
• 24:41
  SPEAKER_02: Uh, Liam, you were gonna uh uh toss out a question there.
• 24:44
  SPEAKER_02: What were your thoughts?
• 24:45
  SPEAKER_00: Yeah, I had a couple of things crossed my mind, but I've been so hyper focused on this conversation.
• 24:51
  SPEAKER_00: It's really, really cool.
• 24:53
  SPEAKER_00: Uh, I didn't want to interrupt anything.
• 24:55
  SPEAKER_00: Gonna kind of rewind a little bit when we were talking about the importance of prompts with content creation and generative AI.
• 25:02
  SPEAKER_00: Uh, Leif, I know you have a lot of experience with content creation or content management, and you really kind of understand the science and the processes behind all of it.
• 25:10
  SPEAKER_00: Um, my question is to kind of get both perspectives from you and Seth.
• 25:15
  SPEAKER_00: I know one of like the big AI misconceptions that a lot of people have is they get worried about job security or like certain things that they're gonna be doing day to day is now oh, irrelevant.
• 25:25
  SPEAKER_00: What do I do instead?
• 25:27
  SPEAKER_00: Now that kind of I guess ties into content creation and things like graphic designers when it comes to you know making AI generated images or like copy for this podcast episode description, for example, we're gonna use that as a tool.
• 25:41
  SPEAKER_00: Do you see kind of like any sort of I guess room in the future when it comes to utilizing generative AI for content creation, um, with like any sort of specialization in like prompt engineering, or I guess like I guess I'd call it prompt people or prompt specialists?
• 25:57
  SPEAKER_00: Do you think there's kind of like room in the content creation or content management space for people that specialize in just being the best prompters ever?
• 26:06
  SPEAKER_00: Um, whether it be for some like an RAG, you know, focused LLM or just like using Chat GPT that might um, I guess, show some sort of space in the tech industry for um growth or money or education, you know.
• 26:22
  SPEAKER_00: Just kind of wanted to get your perspective on that.
• 26:24
  SPEAKER_00: Prompt people.
• 26:25
  SPEAKER_01: Yeah, Liam, that's that's that's already happening.
• 26:28
  SPEAKER_01: Uh there's okay, there is a uh as as we sit here and record uh this podcast, uh, there is a newly posted uh, it wasn't a prompt engineer position, it was a content designer for chatbots or something like that, job description, where that person was going to be entirely responsible for uh feeding the content into the uh the LLM, training the LLM, uh querying and ensuring the accuracy of the responses.
• 27:07
  SPEAKER_01: And they were going to be responsible for uh creating tickets for the engineering team as well as for the content team.
• 27:20
  SPEAKER_01: So there is absolutely going to be that type of work.
• 27:24
  SPEAKER_01: Now, this is an organization that recognized that they needed to avoid uh certain issues, they don't want the LLM producing fictional responses, no hallucinations, or they they want to be able to catch the hallucinations before their customers do.
• 27:45
  SPEAKER_01: And so, yes, there is going to be absolutely room, uh a new a new opportunity.
• 27:54
  SPEAKER_01: And on the other side of that is as content creators, people are going to become more efficient.
• 28:03
  SPEAKER_01: Uh, there is uh an assistant sitting there available to them.
• 28:08
  SPEAKER_01: The blank page is always the hardest one to fill, and the the generative AI systems will certainly help authors and content creators fill in that blank page.
• 28:24
  SPEAKER_01: Maybe it's just with an outline of if they need to create a certain structure of content uh be that is that is known and understood, they can ask for some defaults.
• 28:40
  SPEAKER_01: And maybe that's uh where more specific prompts will need to be written, but it's a start.
• 28:48
  SPEAKER_01: And Seth, you had mentioned earlier sort of the the stacking prompts on top of each other, the queries building on on each other, and that is where we go next.
• 29:03
  SPEAKER_02: Yeah, I would agree, I would agree with that, and and uh Liam is a good question because uh the prompts are really the queries, they're the ways of asking the questions, and those should be aligned with the uh content architecture and the knowledge engineering piece of this, because basically, when we're asking a question of a data source, whether it's an LLM by itself or an LLM with a structured repository, a source of truth, what we're doing is we're we're giving it hints about what entities are important, you know, write a marketing report or you know, you know, uh write a proposal, write a presentation, write an outline for a speech, right?
• 29:47
  SPEAKER_02: There's content types in there, there's topics in there.
• 29:50
  SPEAKER_02: And again, as we start doing this with more of our own content rather than relying on the LLM, those tags are going to be important.
• 29:58
  SPEAKER_02: And aligning those tags with the prompts will be important.
• 30:01
  SPEAKER_02: So the prompt engineering is really query building, right?
• 30:05
  SPEAKER_02: It's really understanding how to ask the question of the LLM or how to ask the question of the repository, leveraging the LLM's ability to normalize that that uh query, that prompt, that that question, right?
• 30:19
  SPEAKER_02: And and thinking of how you're getting to, you know, these are all in multidimensional vector space.
• 30:27
  SPEAKER_02: It's hard to think in multiple dimensions, right, beyond three dimensions, but you know, they're n-dimensional.
• 30:33
  SPEAKER_02: In fact, you know, there are as many dimensions as there are features.
• 30:36
  SPEAKER_02: And so we're talking about, you know, hundreds of thousands, millions, you know, in in a large language model, you know, there could be a billion parameters, right?
• 30:45
  SPEAKER_02: I mean, there are there are the the the whole the parameters of the large language model itself can be multiple billions of parameters, but even a vector representation of a piece of content could have tens of thousands of features, right?
• 30:59
  SPEAKER_02: Because each feature is really metadata.
• 31:01
  SPEAKER_02: And what we're trying to do is navigate this three-dimensional this n-dimensional space to get closest to what that right answer would be.
• 31:10
  SPEAKER_02: And so the prompt is actually being converted into a vector, and then that vector is being compared to other vectors and saying, well, what's the closest vector across these multiple dimensions where I can get the answer that is most suitable to my query?
• 31:28
  SPEAKER_02: And again, those queries can be so complex, they're going to have thousands of dimensions.
• 31:34
  SPEAKER_02: And that's where you know the power of these things is just is vast.
• 31:39
  SPEAKER_02: It's it's incredible.
• 31:40
  SPEAKER_02: And that's why it's so computationally intensive and heavy.
• 31:43
  SPEAKER_02: So the idea of saying, you know, let me let me give all this uh context, right, for a uh for a prompt, you know, you're talking about audience, you're talking about topic, you're talking about voice, you're talking about all these things that could be broken down into metadata and the metadata of the prompt.
• 32:04
  SPEAKER_02: If you say I'm going to build prompts, I need to include certain metadata.
• 32:08
  SPEAKER_02: That metadata needs to be aligned with my content model.
• 32:11
  SPEAKER_02: So I look at it as both sides of saying build the prompt using knowledge engineering uh approaches or align your prompt engineering with your knowledge engineering so that you can make the best of both of those worlds, because otherwise, you know, you're you're you know, you're not able to correctly navigate that space, that that vector space.
• 32:33
  SPEAKER_02: And I know it's hard for people to imagine that, but that's what we're doing.
• 32:37
  SPEAKER_02: You know, we're navigating multiple dimensions, and and those dimensions can be thou, they can be thousands of dimensions, right?
• 32:43
  SPEAKER_02: You it's impossible for the human mind to comprehend, but that's what's happening mathematically with these tools.
• 32:50
  SPEAKER_02: And and as I say, the the more the greater number of parameters in a model, and that's why we say, you know, model has a billion, two billion, whatever billion parameters.
• 32:59
  SPEAKER_02: I don't know what the largest number is right now.
• 33:01
  SPEAKER_02: Do you know, offhand leaf, how big the models are?
• 33:05
  SPEAKER_01: I don't I don't.
• 33:06
  SPEAKER_02: 1.8 billion, I forget what it is.
• 33:08
  SPEAKER_02: So uh in the background, Leem, you can do a search on that.
• 33:12
  SPEAKER_02: Sure.
• 33:12
  SPEAKER_02: What are the parameters for the largest, largest of large language model?
• 33:15
  SPEAKER_02: By the time this gets posted, it will have changed.
• 33:17
  SPEAKER_02: Yeah, exactly.
• 33:18
  SPEAKER_02: But the point here is you know, what is the what is the actual uh and and the greater the complexity and the more the parameters that large language model has, the more complex the queries it can manage and the prompts it can manage.
• 33:31
  SPEAKER_01: But but what's that in some ways it's it's way simpler than that from the user's perspective.
• 33:40
  SPEAKER_01: The user and you you used a lot of language that spoke to information architecture without actually saying information architecture.
• 33:50
  SPEAKER_01: And you also spoke to user experience from the user's experience or from the user's perspective, they're typing in to a text field.
• 34:00
  SPEAKER_01: They have one dimension that they're looking at, and they have no idea what the the navigation or the content structure looks like.
• 34:12
  SPEAKER_01: A chatbots interface is very different than a website's interface.
• 34:20
  SPEAKER_01: A website gives you a number of clues to what the the content structure might be.
• 34:27
  SPEAKER_01: There's navigation and there, you know, headings and texts that labels that you can see.
• 34:36
  SPEAKER_01: There are facets that you might be able to choose.
• 34:39
  SPEAKER_01: None of that exists with a chatbot or a virtual assistant.
• 34:43
  SPEAKER_01: So from a querying perspective, there is nothing exposed to the user and the interfaces that I have seen that give any context to what the corpus structure might be.
• 35:00
  SPEAKER_01: So the user has to ask blindly what is in what is in the content uh the corpus, and the response will give them some more information, which then might trigger a conversation of well, no, no, no, that wasn't exactly what I needed, right?
• 35:19
  SPEAKER_01: And I have to change my question.
• 35:22
  SPEAKER_00: Well, yeah, and I was 175 billion for uh open AI chat GPT.
• 35:29
  SPEAKER_02: But that was, I think, 3.5, maybe.
• 35:32
  SPEAKER_02: Let's see, three GPT three was 175 billion.
• 35:36
  SPEAKER_02: They said that GPT 4 uh possesses trillions, trillions of parameters, trillions.
• 35:43
  SPEAKER_02: So the large language models, that's the number of parameters.
• 35:46
  SPEAKER_02: And again, features are one of the parameters.
• 35:49
  SPEAKER_02: The feature can be thought of as metadata, right?
• 35:51
  SPEAKER_02: It so, so, so Leaf, your point of saying that the user doesn't have any understanding of that structure.
• 35:57
  SPEAKER_02: That's why where I'm making the argument that we do want to build these things with some awareness of those structures, right?
• 36:05
  SPEAKER_02: To say, let's do our prompt engineering that contains certain entities and our content that will contain certain entities.
• 36:13
  SPEAKER_02: And then part of this is um exposing that as a as a dialogue, right?
• 36:19
  SPEAKER_02: As a uh a conversation.
• 36:21
  SPEAKER_02: Point here is that if if I ask an ambiguous question, the LLM has to be able to respond with a clarifying question, just like with faceted search, when we put in, if we type in tools on the Granger site, it's gonna give us hand tools, power tools, this kind of tool, that kind of tool, right?
• 36:41
  SPEAKER_02: It's gonna give us a whole bunch of different parameters where we can clarify.
• 36:45
  SPEAKER_02: That's the question.
• 36:45
  SPEAKER_02: You know, if you walked up to a counter in a in a uh you know, in a hardware store and you said tools, you know, what would they say?
• 36:53
  SPEAKER_02: Okay, what do you want to know about tools?
• 36:55
  SPEAKER_02: Or what kind of tools?
• 36:56
  SPEAKER_02: What are you trying to solve?
• 36:57
  SPEAKER_02: So the same thing has to happen with our chat interfaces, where the chat bot has to be able to ask clarifying questions in order to get context.
• 37:08
  SPEAKER_02: And I don't think anybody's really doing that right now, but I think the idea is to say, you know, ask questions and use those questions to guide the the LLM in where to go in that vector space so that you can leverage facets.
• 37:24
  SPEAKER_02: You can leverage, and you're not maybe you're not doing it explicitly, but you could you could do it explicitly.
• 37:30
  SPEAKER_02: You could ask clarifying questions, which could have you know results from a from a faceted retrieval.
• 37:36
  SPEAKER_02: But I think that it's it's having that awareness of the content and saying, well, we do need some structure of the content.
• 37:42
  SPEAKER_02: You know, we had done that uh study that showed that uh adding metadata and a content architecture to content that's ingested into a vector space is actually more accurate, produces more accurate results, right?
• 37:55
  SPEAKER_02: It was 53, and this is where we're preventing hallucinations by saying, you know, if you don't have the answer from this data source, say I don't know.
• 38:03
  SPEAKER_02: So again, that was something where, you know, the LLM was restricting its query to a specific data source.
• 38:12
  SPEAKER_02: And then within that data source, we had the additional signals, which are essentially features or parameters of the content.
• 38:20
  SPEAKER_02: And that additional metadata, you know, again, can finely tune uh that retrieval.
• 38:26
  SPEAKER_02: But at the same time, we have to think about those questions, right?
• 38:30
  SPEAKER_02: So it's it's it's saying, well, what are people?
• 38:32
  SPEAKER_02: This is why use cases are so important.
• 38:34
  SPEAKER_02: What are people trying to accomplish?
• 38:36
  SPEAKER_02: And if they're trying to get a certain type of information, that has to be in the content model.
• 38:40
  SPEAKER_02: So those prompts need to be engineered or the or the content model needs to be reverse engineered from the prompts, essentially.
• 38:49
  SPEAKER_01: Does that make sense?
• 38:50
  SPEAKER_01: I understand what you're saying.
• 38:52
  SPEAKER_01: And what you know, you you started backing out to you know the the area of my focus, which is the the strategy of it all.
• 39:00
  SPEAKER_01: Like, why are we doing this all in this project in the first place?
• 39:04
  SPEAKER_01: What is it that we're looking to achieve from the from the business or for the business?
• 39:10
  SPEAKER_01: And and being very specific about that uh helps uh define what success looks like for that project.
• 39:18
  SPEAKER_01: So, you know, rather than you know, having a chatbot that will respond to any and all queries, maybe it's uh at first only within uh a specific domain within your content set so that you can understand where the the wiggle room is in the the responses that come from the the chatbot, the LLM.
• 39:44
  SPEAKER_01: So that you can then go back and refine your prompts and uh get more specific with the maybe the training, the curation of the content that goes in.
• 39:55
  SPEAKER_01: Because more than anything, a generative AI project is a content project.
• 40:03
  SPEAKER_01: It is getting the terms aligned, having the taxonomy, what moving things into you know an ontology so those relationships are clearly defined because that all informs the the LLM, which is going to produce a more accurate, better response.
• 40:24
  SPEAKER_02: That's right.
• 40:25
  SPEAKER_02: So I think we have to start with those narrow sets of use cases and a defined corpus and say, let's get this laser focused to solve this specific problem.
• 40:38
  SPEAKER_02: Maybe it's support of a particular product, maybe it's customer service queries around, you know, uh whatever the service offering or whatever the the product suite is.
• 40:47
  SPEAKER_02: But that's where technical documentation comes in, because we can take that technical documentation, ingest it into the LLM with the additional metadata, right?
• 40:57
  SPEAKER_02: Because those become the features and the context, and then use that as a way to query that content.
• 41:03
  SPEAKER_02: And and essentially, you know, right now people are on the other end of the spectrum, right?
• 41:06
  SPEAKER_02: They're just saying, well, let's just, you know, put all our content there and ask it questions.
• 41:11
  SPEAKER_02: Well, that's not going to work, right?
• 41:12
  SPEAKER_02: We have to be more focused, you know, we get to be laser focused on a do a set of use cases, a domain of knowledge, a corpus, and then we need to think about the relationship of the prompts and the prompt engineering with the knowledge engineering.
• 41:26
  SPEAKER_01: I I I I want to focus in on that response a little bit, Seth, because it actually will work.
• 41:33
  SPEAKER_01: Uh, because from which other side will produce a it will produce a response.
• 41:42
  SPEAKER_01: It's only as you get into the details of the response that I think you and I would agree that maybe the response or the answer wasn't accurate or precise enough, or maybe it was even misleading.
• 41:55
  SPEAKER_01: Right.
• 41:56
  SPEAKER_01: But the the technology is going to produce a response.
• 42:00
  SPEAKER_01: And so at one level, it will look like success.
• 42:04
  SPEAKER_02: Right.
• 42:04
  SPEAKER_02: But it but again, it comes back to those use cases because you need a gold standard.
• 42:08
  SPEAKER_02: You need to say, how do I judge the the success?
• 42:11
  SPEAKER_02: How do I judge the accuracy?
• 42:13
  SPEAKER_02: Because uh I was talking to someone the other day at a life sciences firm who said, you know, we asked the question of what's the lunch expense policy?
• 42:22
  SPEAKER_02: And it said, you know, uh$180,000, right?
• 42:26
  SPEAKER_02: For lunch, right?
• 42:27
  SPEAKER_01: And I could have I could I could get lunch for good lunch for that.
• 42:31
  SPEAKER_02: Yeah.
• 42:31
  SPEAKER_02: And and it was, and it was, and it was saying for Argentina or someplace, right?
• 42:35
  SPEAKER_02: It was not necessarily saying for their location.
• 42:37
  SPEAKER_02: And then no matter how they asked the question, you came back with that uh ridiculous or erroneous answer.
• 42:42
  SPEAKER_02: So you're right, it will produce a result.
• 42:44
  SPEAKER_02: It's not producing satisfying results, it's not producing trustworthy results, and sometimes it's not producing accurate results.
• 42:50
  SPEAKER_02: So that's what we need to do.
• 42:51
  SPEAKER_02: Uh Liam, you had another question.
• 42:53
  SPEAKER_00: Yeah, I wanted to talk more about content management and strategy with generative AI, because this to me has just been like a really, really interesting topic lately, the past couple of months.
• 43:03
  SPEAKER_00: So when it comes down to like uh, excuse me.
• 43:06
  SPEAKER_00: So let's say uh fast forward into the future, you realize that like the entire internet is just every single company using generative AI to make things, make stories, write articles, make photos.
• 43:19
  SPEAKER_00: And then it comes down to like legal or copyright concerns and considerations.
• 43:25
  SPEAKER_00: Who really does own that?
• 43:26
  SPEAKER_00: And I guess like that's kind of like how I think about it.
• 43:28
  SPEAKER_00: And I've had a I had a conversation with a friend in one of my IT classes about this where he was like, Hey man, if Da Vinci made a machine that painted the Mona Lisa, was it Da Vinci that painted the Mona Lisa or was it the machine that painted the Mona Lisa?
• 43:43
  SPEAKER_00: You know what I mean?
• 43:44
  SPEAKER_00: And uh I just wanted to kind of get your perspective or hear what you have to say.
• 43:51
  SPEAKER_01: That is not my area of expertise.
• 43:54
  SPEAKER_01: I don't, I'm not certain that I would feel comfortable we waiting into.
• 44:00
  SPEAKER_01: The the waters of copyright.
• 44:02
  SPEAKER_01: I will leave that to other folks more trained and you know thinking about those issues.
• 44:10
  SPEAKER_01: No doubt it is one that we should be concerned about, uh, very much so, both in the sense of what is the legality of uh the text that's being generated, who owns that, uh, but also uh from the the ethics perspective, you know, is the the responses that are being generated uh ethical to uh and any and everybody?
• 44:39
  SPEAKER_01: And then, you know, lastly, you know, are they internally consistent uh with uh each each other within a content set?
• 44:50
  SPEAKER_01: And we're already seeing instances where uh organizations have put chatbots out there in efforts to reduce their support calls or uh have another avenue of interacting with the organization.
• 45:08
  SPEAKER_01: And depending on how successful they were at their training and their curation, uh sometimes they have failed spectacularly.
• 45:19
  SPEAKER_01: Uh, one organization uh I uh read about had a 12-fold increase in their support calls.
• 45:28
  SPEAKER_01: Wow, that that chatbot didn't stay up but for two weeks before it was uh taken off the site for obvious reasons.
• 45:38
  SPEAKER_01: And uh relatively recently there was a a well-publicized uh uh situation where uh a user had used a uh the chatbot on a company's website to ask about a policy, and it received uh the the person received uh an answer uh and had made a decision based on the response from the chat bot, only to learn later that the chat bot had produced an inaccurate response and that it the the information from the chat bot conflicted with information on the website.
• 46:23
  SPEAKER_01: And uh the person sued the organization, and the organization ended up losing because the judge uh ruled that uh how is the person supposed to know if interacting with the chatbot on the website?
• 46:38
  SPEAKER_01: Well, it the onus shouldn't be on the person to go find every page to verify that there's no contradicting information.
• 46:46
  SPEAKER_02: Exactly, exactly.
• 46:48
  SPEAKER_02: Hey, um uh we are right at the top of the hour.
• 46:51
  SPEAKER_02: This has been fantastic.
• 46:52
  SPEAKER_02: I would love to see that that case study or that that example if you want to send me that information.
• 46:57
  SPEAKER_02: But um Leaf, thank you so much for uh joining us today.
• 47:01
  SPEAKER_01: We really appreciate it.
• 47:03
  SPEAKER_01: Seth and Liam, this has been uh the time went by so quickly.
• 47:07
  SPEAKER_01: What a joy.
• 47:09
  SPEAKER_00: It was a pleasure.
• 47:10
  SPEAKER_02: All right, and uh thank you to you to all the folks that are listening.
• 47:15
  SPEAKER_02: Thank you, uh Carolyn, and of course, Liam, uh, for your for your efforts.
• 47:19
  SPEAKER_02: And we will see you next time.
• 47:22
  SPEAKER_02: Thanks, everyone.
• 47:25
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 47:29
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 47:35
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and CMS Wire.
• 47:41
  SPEAKER_00: To learn more, visit early.com.
• 47:43
  SPEAKER_00: That is E A R L E Y dot com.
• 47:48
  SPEAKER_00: Thanks for tuning in.