Latest / Future of Work Tech with Fexingo: Remote Tools, AI Productivity, and Workplace Software / How AI Is Automating Your Internal Knowledge Tagging
Transcript
- Lucas: So there's this question a lot of companies are waking up to right now. They've got tens of thousands — sometimes hundreds of thousands — of internal documents, wikis, chat logs, and nobody has ever tagged them. No metadata. No taxonomy. It's just a huge pile of text. Luna: Right, and that means search is basically useless. You type in something, get back a thousand results, none of them the right one. Lucas: Exactly. And traditionally, solving this means hiring a team of librarians or information architects to go through everything manually. It's slow, expensive, and by the time you're done, more content has piled up. Luna: So where does AI fit? I mean, we've talked about AI automating a lot of things on this show, but tagging internal documents — that sounds like a perfect use case. Lucas: It really is. And there's a great example from a mid-size B2B SaaS company I was looking at recently. They had about fifty thousand internal documents — everything from product specs to onboarding guides to legal policies. They used an open-source language model to tag every single one in under three hours. Luna: Three hours? That's wild. And what did it cost? Lucas: Around three hundred dollars in compute. Compare that to what a team of, say, five contract librarians would charge. They estimated about seventy-five thousand dollars for the same work, and that's not counting the months it would take. Lucas: And just to be clear, we don't run ads on this show. The reason we can talk about specific numbers like that without a sponsor's influence is that listener support keeps it ad-free. If that kind of independent analysis is valuable to you, there's a link: buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep the conversation focused on what actually works. And speaking of what works, let's talk about accuracy. Fifty thousand documents in three hours sounds great, but how accurate were the tags? Lucas: That's the key question. In this case, they ran a validation sample. They had a human team review about five hundred documents and check the ai generated tags. They found about 92 percent precision — meaning when the AI assigned a tag, it was correct 92 percent of the time. Luna: 92 percent is solid. But what about recall? Did it miss tags that should have been there? Lucas: Recall was a bit lower — around 85 percent. So it was missing about 15 percent of relevant tags. That's a trade-off you have to manage. For routine content, that might be fine. But for legal or compliance documents, you'd want a human to review. Luna: Right, because an AI hallucinating a tag on a contract could have real consequences. Imagine it tags something as 'approved' when it's actually still under review. Lucas: Exactly. And that's where the human-in-the-loop model comes in. You let the AI do the heavy lifting — tag everything, surface the low-confidence predictions — and then a person reviews only those edge cases. That hybrid approach is what most companies are adopting. Luna: So it's not about replacing the librarians entirely. It's about letting them focus on the tricky stuff while the AI handles the bulk. Lucas: Precisely. And the cost savings are still enormous. Even with a human review of, say, 10 percent of the documents, you're looking at maybe ten thousand dollars total instead of seventy-five thousand. That's a six-figure saving for a lot of teams. Luna: And what about the model itself? You said open source. Which one? Lucas: They used a fine-tuned version of Llama 2, specifically a 13-billion-parameter model. That's small enough to run on a single GPU, but big enough to capture nuanced categories. They trained it on a few hundred manually tagged examples from their own documents. Luna: So the company needed some labeled data to start. That's a barrier for a lot of teams. Lucas: It is. But you can start with as few as fifty to a hundred examples. And if you don't have any, there are zero-shot approaches using models like GPT-4 or Claude, where you just describe the categories you want. That's more expensive per document, but still way cheaper than manual tagging. Luna: Let's talk about categories. What kind of tags are we talking about? Just broad topics like 'marketing' or 'finance'? Or more granular? Lucas: In their case, they had a three-level taxonomy. Top level was department: engineering, marketing, sales, legal. Second level was document type: policy, guide, report, template. Third level was specific project or product. So a document could be tagged 'Engineering — Code Review Guide — Acme Project'. Luna: That's a lot of granularity. And the AI handled all three levels? Lucas: Yes. They trained it to output a JSON object with those three fields. And they set confidence thresholds. If the model's confidence was below 70 percent on any field, it flagged that document for human review. That's how they managed the precision-recall balance. Luna: Smart. So the system knows what it doesn't know. That's a nice safeguard. Lucas: It's a pattern I'm seeing more and more across enterprise AI. The models aren't perfect, but if you build in these confidence gates, you get the efficiency of automation without the risk of silent failures. Luna: What about images and audio? Most internal knowledge isn't just text — it's slide decks, screenshots, recorded meetings. Lucas: That's the next frontier. And we're starting to see multimodal models that can tag images and audio with similar accuracy. For example, you can feed a slide deck image into a vision-language model and get keywords like 'Q3 revenue chart' or 'product roadmap Q4'. Luna: But that's got to be more expensive, right? Processing images is compute-heavy. Lucas: It is. But the cost is dropping fast. A year ago, tagging an image might cost a few cents. Now it's more like a fraction of a cent. And for audio, you transcribe first, then tag the transcript. That adds a step, but the overall pipeline is getting streamlined. Luna: So where do you see this going in the next year or two? Will manual tagging become obsolete? Lucas: Not entirely. For highly sensitive documents — think legal contracts, HR records, financial audits — you'll always want a human in the loop. But for the vast majority of internal content, I think we'll see a default where AI does the tagging and humans only review exceptions. Luna: That would free up a lot of people for higher-value work. Instead of tagging documents, they're analyzing the tags, finding patterns, improving search. Lucas: Exactly. And that's the real win. It's not just saving money — it's making your institutional knowledge actually findable. You stop reinventing the wheel because someone can't find the old project retrospective. That kind of efficiency compounds over time. Luna: Alright, I'm sold. Next step: where do I start if I want to try this for my own team? Lucas: Start small. Pick a single folder with maybe fifty documents. Define five to ten categories you care about. Use a zero-shot model like Claude or GPT to tag them. Then check the tags manually. See where it works and where it fails. That'll tell you whether a custom fine-tuned model is worth the effort. Luna: And the open-source route — is it accessible to teams without a machine learning engineer? Lucas: It's getting easier. There are platforms like Hugging Face that offer hosted inference endpoints. You can use a pre-trained model and just send it text. No GPU needed on your end. But if you want to fine-tune, you'll need some scripting skills or a data scientist friend. Luna: So for a typical team, the zero-shot API route is probably the fastest path to proof of concept. Lucas: Absolutely. And the cost is so low that the experiment itself is almost risk-free. Even if you tag a thousand documents, you're talking maybe ten or twenty dollars. That's a cheap way to learn whether AI tagging works for your content. Luna: I like that — low risk, high potential upside. Sounds like a good Friday afternoon project. Lucas: Exactly. And if it works, you can scale it from there. One folder becomes ten, then a hundred. Before you know it, your entire knowledge base is tagged and searchable. And you didn't have to hire a team of librarians. Luna: Alright, I'm going to try that with our own internal docs. See you next time. Lucas: Sounds good. We'll check back in a few weeks and see how it went.