Latest / Microsoft Cloud IT Pro Podcast / Episode 402 – Declarative Agents with Andrew Connell
Transcript
- 0:03Welcome to episode 402 of the Microsoft Cloud
- 0:06IT Pro podcast recorded live 05/16/2025.
- 0:11This is a show about Microsoft three sixty
- 0:14five and Azure from the perspective of IT
- 0:16pros and end users, where we discuss a
- 0:18topic or recent news and how it relates
- 0:20to you.
- 0:21This week, Ben and Scott have Andrew Connell
- 0:24or AC
- 0:25back on the show to loop back into
- 0:27some topics from episode 400 around custom agents,
- 0:31but from a bit of a developer perspective.
- 0:33They discuss some differences
- 0:35besides price in the various billing models and
- 0:38limitations
- 0:39around creation of the semantic index for Copilot.
- 0:42They also discuss how Copilot uses the semantic
- 0:45index and access control lists for security when
- 0:47retrieving your content. Another topic of conversation is
- 0:51potential issues you may encounter around large files,
- 0:55ASPX pages, and missing metadata when it comes
- 0:58to Copilot
- 0:59responses, as well as potential workarounds by using
- 1:01Azure AI search.
- 1:03Finally, they wrap up the show by talking
- 1:05about some topics you may want to pay
- 1:07attention to from this week's Microsoft Build Conference.
- 1:11Let's dive into the show.
- 1:15For all of you listening, it's not just
- 1:17Scott Knight today. There's a new I would
- 1:19say a new voice. It's kind of an
- 1:21old voice. I don't know how many times
- 1:24Wow. Just
- 1:25digging that lightning. Out swinging today. It's a
- 1:28Friday morning.
- 1:29Said, we have a popular repeat guest. Andrew
- 1:32Connell's back on the show. But then, no,
- 1:34you just went to the end of it.
- 1:36Just said it all yeah. I shoulda had
- 1:37you kick it off then, Scott.
- 1:40Now I'm all we're we're all off today.
- 1:42This is what happens on a Friday morning
- 1:43when it's been a long week. But, yeah,
- 1:46we do have Andrew Connell back on the
- 1:47show to talk more about CoPilot agents. So
- 1:50I don't know how many times we've had
- 1:51you on now, Casey. It's probably like four
- 1:53or five. Has it been that many? I
- 1:55know it's, like, one or two. I didn't
- 1:56know I didn't know if it's been I
- 1:57don't know. But we're go hey. It's a
- 1:59long time listener. Let's just say second or
- 2:01third or fourth time caller. Yeah. Exactly.
- 2:04A popular that popular caller on our show
- 2:07that just keeps coming back.
- 2:09But, no, we did, like, episode
- 2:11I think it was was it 3 no.
- 2:13It was 400 was the Copilot agent once.
- 2:16And you were listening to it, and you
- 2:17sent us a message and said, hey, Ben.
- 2:19Let's talk about some more stuff around Copilot
- 2:21agents that
- 2:22you've come to discover. And I would say
- 2:24even from a developer's perspective, like, Scott and
- 2:26I are very IT pro, which everybody knows,
- 2:29and you are very developer. But I think
- 2:30that also gives you a new perspective sometimes
- 2:33on how some of the stuff works under
- 2:34the covers. You uncover some stuff. You're like,
- 2:37Ben and Scott, let's talk about some other
- 2:39stuff that
- 2:40relates to that. So we're gonna take a
- 2:42little bit of a journey adding some
- 2:44color, some additional information to what we discussed
- 2:47on episode 400. So if you haven't listened
- 2:50to that one, you may even wanna listen
- 2:51to that one first. And then even maybe
- 2:53dive into some of the development stuff with
- 2:55Copilot agents and how you can expand Copilot
- 2:57agents even more from
- 2:59if you have access to a developer, if
- 3:01you are a developer listening to the show.
- 3:03Yeah. That sounds great. But yeah. You guys,
- 3:05first of all, like, congratulations
- 3:07on 400 episodes doing this for eight years.
- 3:10That's a huge accomplishment.
- 3:12It is I've done eight I've done a
- 3:13podcast for that long and that many episodes,
- 3:15and I will I know how big how
- 3:17much effort it is and how how much,
- 3:19like, staying power you've done with it. So
- 3:21huge kudos to you guys for doing this.
- 3:23I remember when you started this, and it's
- 3:24like it's really I I was listening to
- 3:26this on your episode 400 on the way
- 3:28to the airport, and I, like, did a
- 3:29note to myself. I'm like, oh my god.
- 3:30Gotta congratulate these guys. That's freaking awesome. I
- 3:33didn't realize they were at 400. And then
- 3:34I listened to the rest of the episode,
- 3:35and I was like, I'm gonna get to
- 3:36the airport, and I gotta send a loom
- 3:37over to to to you guys and be
- 3:39like, I got some stuff I wanna talk
- 3:40about.
- 3:41Yeah. Well, thanks. And thanks for encouraging us
- 3:44to start it because I was gonna say
- 3:45you're one of the catalysts. So Yeah. There
- 3:47you go. You, like, propelled us into doing
- 3:50this. You're like, you really should do it.
- 3:51And then you got me to go chase
- 3:52down Scott and convince Scott to do it
- 3:54with me. So thank you. And CJ as
- 3:57well. You guys were both an inspiration to
- 3:59starting it episode one eight years ago. I
- 4:02missed doing it, and I'm looking forward to
- 4:04starting another one by the time this goes
- 4:06live. I think I might have it might
- 4:08be live by then, so we'll see. Oh,
- 4:10alright. Keep our eyes out for us. Yeah.
- 4:12Put a link. We'll throw it in the
- 4:13show notes. Alright. Sounds good. Yeah. Where do
- 4:15you wanna start? Some of the stuff that
- 4:17we talked about. I don't know. There were
- 4:19there's kind of a couple things. I don't
- 4:21know if we wanna start with some of
- 4:22the stuff around deploying it, different models for
- 4:25pricing it, or if you wanna start with
- 4:27some of the other topics and what's going
- 4:28on in the background. Where if you were
- 4:30gonna explain this to somebody, where would you
- 4:32start that journey? Doctor. Let me build off
- 4:34what you guys said in episode 400 because
- 4:36there was a few things that I thought
- 4:37were not that you missed anything, but there
- 4:40was a few extra things that I've seen
- 4:41that when I talk to a customer
- 4:43that it's a it's like a you should
- 4:45know this
- 4:46before you go into things. So one of
- 4:49them was you guys were talking a little
- 4:50bit about the billing, and there was, like,
- 4:51the difference between $30 per user per month,
- 4:54or do I spend $20.200
- 4:56for 25,000 messages? Doctor. Yep. Doctor. And or
- 4:58do I go with the consumption model and
- 5:00pay a penny per message? Of course, all
- 5:02USD pricing.
- 5:03Sorry. All my numbers are USD pricing. I'm
- 5:05sure you could buy this outside of outside
- 5:06of The US. Doctor. Yeah. Doctor. There's one
- 5:08thing that I see when people do that
- 5:09that is a huge blind spot that a
- 5:11lot of people don't realize. And if they
- 5:13go with the consumption model first,
- 5:15they will have this copilot sucks type attitude
- 5:18to it because it's like, I thought it
- 5:19was supposed to be grounded in all my
- 5:20data. So there's there's two aspects to it.
- 5:22So first of all, one thing to understand,
- 5:24and I think you guys talked a little
- 5:25bit about this, is that when Microsoft says
- 5:27that it is grounded in your data and
- 5:30they always point to Microsoft Graph, that is
- 5:32kind of true. I mean, it's not incorrect.
- 5:36But to be a little bit more technical
- 5:37about it is, like in SharePoint, you put
- 5:40content in SharePoint, and then for search to
- 5:42work in SharePoint, there's an indexer that creates
- 5:45an index so so that when you do
- 5:46execute a keyword search, it doesn't look at
- 5:48the content in SharePoint. It looks at the
- 5:50index, finds the results, and it gives you
- 5:52gives you the results and it's links to
- 5:53the content in SharePoint. Yep. So that's how
- 5:56Copilot is grounding your information. So it's not
- 5:59it's doing a similar model. What it does
- 6:01is it gets data from Microsoft Graph about
- 6:04you and your organization. So that's data from
- 6:06SharePoint, data from
- 6:08chat messages, Teams transcripts, meeting transcripts, all that
- 6:11kind of stuff. And what it does is
- 6:13it does another kind of indexing called a
- 6:15semantic index, which is not keyword index, but
- 6:18a semantic index is more like let's just
- 6:20say, AI based index that is a little
- 6:22bit more these words are closer to these
- 6:24words, and it's just better for, like, AI
- 6:26based queries. What Microsoft is doing is that
- 6:28when they look at that content in graph,
- 6:30when they do a semantic index of it,
- 6:32they are storing that in what's called the
- 6:34semantic index. So, that kind of sits on
- 6:36top of Graph so that when Microsoft when
- 6:39you run a prompt
- 6:40against Copilot,
- 6:41Copilot is looking up content from the semantic
- 6:45index to get content that's relevant to you
- 6:48and your organization
- 6:50to ground
- 6:51the response
- 6:52in your data and not just like the
- 6:54foundational knowledge that the model has that like
- 6:56ChatGPT
- 6:57can leverage from, like, the the the training
- 6:59from the different models. Okay. So Office three
- 7:02sixty five search uses search index. Copilot uses
- 7:04semantic index. You've got these two indexes.
- 7:07That's correct. Yeah. And and and it's important
- 7:09to, like, call out, like, semantic index is
- 7:11technically generated with information from the graph. So
- 7:14you're, like, layers of layers in here. And
- 7:17and I think it gets a little confusing
- 7:19to customers because
- 7:20we maybe throw out terms like AI training,
- 7:24AI fine tuning, AI inferencing. Like, the reality
- 7:26is most folks aren't out there doing, like,
- 7:29AI training typically in the terms you're hearing
- 7:31maybe, like, I don't know, me, like, Band
- 7:33Aid about. Like, there there's only so many
- 7:35open AIs and anthropics and and things out
- 7:38there that are training these foundational models.
- 7:40So more and more, I think what I
- 7:42see in in my day to day is
- 7:44kind of folks who are
- 7:46leveraging existing models, and they're effectively
- 7:50into
- 7:51some type of augmentation, be it rag that's
- 7:53out there,
- 7:54be it leveraging systems
- 7:56like m three sixty five Copilot
- 7:58that then build on graph, build on semantic
- 8:01index, but then can also do rag and
- 8:03things like that for further grounding, like contextually
- 8:06in the moment versus, oh, hey. I was
- 8:08able to fine tune this model or push
- 8:11out this specific piece
- 8:13of organizational
- 8:13knowledge and and kind of do a little
- 8:16mini retraining run
- 8:17along the way. So it's all these layers
- 8:19upon layers in there. And it's funny to
- 8:21me how it all comes back to, like,
- 8:22the old days of SharePoint and, like, information
- 8:24architecture. Like, how did you organize this stuff?
- 8:26How did you permission it? How how did
- 8:28that look? And then you and then we're
- 8:29back to the same thing of, like, now
- 8:31it's not just figuring out, like, hey. Why
- 8:33did that search result surface in my search
- 8:35index because the permissions were wrong on that
- 8:37SharePoint site or that list or whatever it
- 8:39was. But now it's, oh my gosh. Where
- 8:41else was that thing potentially off the rails?
- 8:43Oh, over here in Teams, because it was
- 8:46actually in a Teams message, but it was
- 8:48in a group Teams message that I had
- 8:50going with some other folks that were out
- 8:52there. And because The Graph is able to
- 8:53surface that information,
- 8:55all of a sudden, you end up in
- 8:56this
- 8:57in this little bit of a a weird
- 8:59spot. And then you combine that with the
- 9:00fuzziness of I I don't know if you
- 9:02ever do this. I I go into teams
- 9:04all the time just or Copilot the app
- 9:06or Copilot in teams, and I say, hey.
- 9:09Search my messages for the last two weeks
- 9:11or the last six months and
- 9:14summarize blah blah blah that I discussed with,
- 9:16you you know, these people. Well, you can
- 9:18tell Copilot that it doesn't actually know how
- 9:21to execute a time based search based on
- 9:23your messages. So, like, what it pulls back
- 9:25could be completely random, could be outside the
- 9:27boundaries of that. It was all just what
- 9:29was available to it in in its grounding
- 9:31knowledge. Doctor. Right. Right. And I like, the
- 9:33way I like to the way I explain
- 9:34this to customers
- 9:35is that, like, think about if you are
- 9:37doing your taxes for your business, right? If
- 9:39you sit down, you have a question for
- 9:41an accountant, that the accountant is somebody you
- 9:43just you just hired, they have all of
- 9:45the knowledge they've learned from being trained as
- 9:48an accountant when they were in college, right,
- 9:50and just factual stuff on how QuickBooks works.
- 9:52QuickBooks works. They're gonna give you an answer
- 9:54to whatever your problem or whatever your question
- 9:56is. That's like using ChatGPT.
- 9:57What's using Microsoft three sixty five Copilot is
- 10:00he has that information or she has that
- 10:02information, but they also have all of your
- 10:04previous tax returns, and there are have access
- 10:07to your QuickBooks. So now they have context
- 10:09about you that they can add to the
- 10:12actual question. That's what the semantic index is.
- 10:14It's context about you and your organization.
- 10:17Technically, there's two semantic indexes. There's one for
- 10:19users and there's one for organizational data. So,
- 10:22but Copilot figures out which one it's gonna
- 10:24go through and grab. So, there's a few
- 10:26things about that though that the semantic index
- 10:27has some interesting
- 10:29aspects to it that people need to understand
- 10:31I think that really helps in understanding why
- 10:34Copilot can or can't do something. One of
- 10:36those things is around the billing. So, I
- 10:38talked about a second ago the difference between
- 10:40you can start with $30 per user per
- 10:41month. Doctor. Yep. Doctor. It's like the all
- 10:43you can eat plan, then there's the 25,000
- 10:45messages for $200
- 10:47a month, or it's paid per the message
- 10:50down on the consumption or metered plan. If
- 10:52you're a brand new customer to Copilot and
- 10:55you decide, I'm going to start this a
- 10:57little bit easy. I don't wanna do the
- 10:59$30 per user per month with a twelve
- 11:01month commitment, so one license is a $360
- 11:03commitment for a year. If you say, I
- 11:06don't wanna do that. Instead, I wanna do
- 11:08the consumption model and kind of ease my
- 11:09way into it. Here's the thing that people
- 11:11don't realize, and Microsoft does not document very
- 11:14well, if at all. You don't get a
- 11:16semantic index until there is at least one
- 11:20active
- 11:21Microsoft three sixty five Copilot license, the $30
- 11:24per user per month. So if you have
- 11:2610 employees and you want them using Copilot,
- 11:28if you just get the consumption model, all
- 11:31Copilot has is the all Copilot has access
- 11:34to is basically the same stuff that you
- 11:36get from ChattGPT, just the foundational knowledge from
- 11:39the training model. If you're a developer and
- 11:41you go through and extend it and do
- 11:42things like what you guys talked about, like
- 11:44adding actions to it, where you can implement
- 11:46the the RAG pattern, the retrieval of meta
- 11:48generation pattern where you where Copilot can look
- 11:50at your query or your prompt and say,
- 11:52oh, I need this extra data from this
- 11:55other endpoint over, like, in Dataverse. I can
- 11:57pull that stuff in, but it doesn't have
- 11:59any it won't have any knowledge about your
- 12:01chats, your your transcripts,
- 12:04your calendar, your meetings, stuff like that. It
- 12:06won't have that context because the semantic index
- 12:08does not exist. It is not until
- 12:10you decide to spend, I will have one
- 12:13person get that $30 per user per month
- 12:15of my 10 employees, and then I'll do
- 12:17consumption, and it completely changes the experience for
- 12:20everybody that has that now has that consumption
- 12:23model in your organization. So now it's like
- 12:25a lot of people look at this and
- 12:26go, like, Copilot sucks. It's no better than
- 12:28ChatGPT. It doesn't have any it's not grounding
- 12:30any of my any of its knowledge and
- 12:31stuff that's with my organization. It's because it
- 12:33doesn't have the semantic index, the trigger to
- 12:35create it, one act of life. So
- 12:38that's bizarre. So because I've never tried this
- 12:40with consumption models. So if I went in,
- 12:42bought one of the consumption model or the
- 12:43message pack, went into, like, Teams chat Copilot
- 12:46and said, go give me this information about
- 12:48my email or people I email or documents
- 12:50in SharePoint, it's not gonna return any of
- 12:52that. I can't speak for certain about the
- 12:5425,000
- 12:55the message packs. Okay. But I can say
- 12:57on the consumption model, yes, that is correct.
- 12:59The penny per message one. The the penny
- 13:01per message one does not turn on a
- 13:04semantic index. The thing that turns it on
- 13:06is the the $30 per user per month
- 13:09license. A message pack might also do that.
- 13:12I think it does, but I can't be
- 13:14certain on that. Again, Microsoft doesn't do a
- 13:15good job of documenting this. You gotta find
- 13:17the right person and catch them at the
- 13:19right time and and get the answer. I
- 13:21have not that part I haven't I don't
- 13:22have a solid answer on. My belief is
- 13:25that a message pack does also do that
- 13:27because a message pack is also gonna have,
- 13:28like they also talk about if you have
- 13:30a message pack, it's gonna be, like, fifteen
- 13:31fifteen messages just to go through and ground
- 13:33data because that's how much a query is
- 13:34gonna cost to the semantic index. Right. So
- 13:37I I think so, but I'm not certain.
- 13:40And it's I'm assuming maybe. Again, this is
- 13:42not anything that's documented having to do with
- 13:45that cost. If you're paying $200 a month,
- 13:47that probably helps to offset the cost of
- 13:49building the semantic index. $30 a month, same
- 13:52thing. If you're just paying a penny per
- 13:53message, you're like, how do we recover the
- 13:55cost of having the semantic index sitting out
- 13:57there because
- 13:59you're only paying a penny per message? I
- 14:01that would be my speculation as to maybe
- 14:03the rationale, but I agree with you. Like,
- 14:05it should be right
- 14:07in that pricing table of, with this, you
- 14:09do not get a semantic index or grounded
- 14:11in your data with these two. Like, just
- 14:14give us a table and show us that.
- 14:16I completely agree with you on that point.
- 14:18That part to me is very frustrating. That
- 14:20was not it's not as clear.
- 14:21One of the other reasons why I say
- 14:22I think it's only limited to the consumption
- 14:24model is that this wasn't known until the
- 14:26consumption model was announced
- 14:28and when it wasn't even really known at
- 14:31that time, but that'severything kind of points back
- 14:33to that being like the big deal. So,
- 14:34Ithat's why I think the message pack does
- 14:36have it, but again, I'm not a %
- 14:37certain. Got it.
- 14:42Do you feel overwhelmed by trying to manage
- 14:44your Office three sixty five environment? Are you
- 14:47facing unexpected issues that disrupt your company's productivity?
- 14:50Intelligink is here to help, much like like
- 14:52you take your car to the mechanic that
- 14:54has specialized knowledge on how to best keep
- 14:56your car running, Intelligink helps you with your
- 14:59Microsoft cloud environment because that's their expertise.
- 15:02Intelligink keeps up with the latest updates in
- 15:04the Microsoft cloud to help keep your business
- 15:06running smoothly and ahead of the curve. Whether
- 15:09you are a small organization with just a
- 15:11few users up to an organization of several
- 15:13thousand employees,
- 15:14they want to partner with you to implement
- 15:16and administer
- 15:17your Microsoft cloud technology.
- 15:20Visit them at inteligync.com/podcast.
- 15:24That's intelligink.com/podcast
- 15:31for more information or to schedule a thirty
- 15:33minute call to get started with them today.
- 15:35Remember, Intelligink focuses on the Microsoft cloud so
- 15:39you can focus on your business.
- 15:43I'm curious. This is not a question that
- 15:45we necessarily talked about. But I know, like,
- 15:47with sensitivity labels, you're going in and starting
- 15:49to exclude SharePoint sites from Copilot, that type
- 15:52of stuff. It also affects SharePoint search, but,
- 15:55yeah, Copilot and SharePoint search are pulling from
- 15:57two different indexes.
- 15:58Is it because the semantic index is still
- 16:01using the search index
- 16:02to build it, or is it just because
- 16:04some of the functionality
- 16:06is the
- 16:07same? I think it's the functionality is the
- 16:09same. I don't know if they're also using
- 16:11SharePoint as a way to populate the semantic
- 16:13index. They they may be, but I don't
- 16:15I don't think I got it. You don't
- 16:16know or looked into that why that's the
- 16:18case. Well, no, and I and I also
- 16:20don't think that they would be using the
- 16:21keyword indexing for that, for the search index
- 16:24that they've created for doing like SharePoint search
- 16:26or three sixty five search. Right. I think
- 16:27they would actually be going to the content,
- 16:29but that that actually brings up another interesting
- 16:31aspect too that that I that I thought
- 16:33was something that also catches people. And you
- 16:35so you guys, I think there's a little
- 16:36bit of a question around
- 16:38security and permissions. So like if like who
- 16:41has access to the content? Does Copilot have
- 16:43access? Does Copilot and the user have access?
- 16:45Like how does that work? When it comes
- 16:47to the semantic index, it's the same as
- 16:49think of the same way as how SharePoint
- 16:51works. So when something goes into the semantic
- 16:53index, you have the ability to add an
- 16:56ACL to it, an access control list, the
- 16:58same way we do with, like, SharePoint search.
- 17:00So when something is indexed, it has an
- 17:03ACL
- 17:04attached to it or a collection of ACLs,
- 17:06and that collection says
- 17:08these people or these groups
- 17:11have access or do not have access to
- 17:13this. So when Copilot does a query against
- 17:17the semantic index,
- 17:18Copilot's making the query, but it's making that
- 17:20query
- 17:21on behalf of the user that submitted the
- 17:24prompt. Okay. So it's getting so the data
- 17:26that Copilot gets back from semantic index is
- 17:28security trimmed for the user who issued the
- 17:30prompt. The same it Got it. Basically the
- 17:32same way how SharePoint search works. If I
- 17:34don't have access to a document and I
- 17:35do a search, I won't see it. You
- 17:36won't see it. It's just that the part
- 17:38that's me is now that is Copilot is
- 17:41doing it for me. So another question then,
- 17:44how quick is the semantic index? Because like
- 17:45SharePoint search index, right, you can have the
- 17:47continuous index on so that when you change
- 17:50security,
- 17:51it's I mean, let's be honest. It's not
- 17:53always instantaneous because it still has to re
- 17:55index the content to re pull in ACLs
- 17:57when you change security to it. Yeah. Semantic
- 17:59index,
- 18:00any ideas? I think it's kind of the
- 18:02same delay as, like, how long it takes
- 18:04to share point to get in. And the
- 18:05other part is, like, how long is it
- 18:06how how fast is it actually querying that
- 18:08when you issue a prompt to Copilot and
- 18:10it's like dot dot dot Copilot's thinking. Yeah.
- 18:12That's how fast it is.
- 18:14That's one of the reasons why it's like
- 18:15it's going to the semantic index. It's pulling
- 18:17some data back. It's going well, first, it
- 18:18goes to LLM, figuring out, like, what is
- 18:20the user's intent? And that tells it, Oh,
- 18:22I need to go over and find out
- 18:23find calendar message find calendar invites or transcripts
- 18:26or emails.
- 18:28It grabs that. It uses that as part
- 18:30of like it says that it's grounding that
- 18:31knowledge, but it it is. But that's actually
- 18:33pulling data to give it a like a
- 18:36a implementing the rag pattern on the fly.
- 18:37So, it's actually CopaD's going to the LLM
- 18:40multiple times, multiple round trips. So, that's one.
- 18:44So, another one though that's actually big is
- 18:46so that's one about the the whole thing
- 18:47about the the semantic index getting created Doctor.
- 18:50Yep. Doctor. And then also the content that's
- 18:51going in. Here's another
- 18:53I I'm gonna call it a limitation,
- 18:55but make sure you listen to the whole
- 18:56part of this because there's like a big
- 18:58story here. Doctor. Okay. Doctor. So when let's
- 19:00say I put a 300 page PDF into
- 19:03a SharePoint document library,
- 19:05it's getting indexed into the semantic index, but
- 19:09only
- 19:09the first percentage
- 19:11of that file. And now, I'm going to
- 19:14use some numbers here, but these numbers are
- 19:16always getting bigger. And so, just take this
- 19:19as an example, and this was true months
- 19:21ago, but I'm sure that when the time
- 19:24this episode comes out, when Build has happened,
- 19:26they would have made another announcement, so the
- 19:28numbers may be bigger. So, when if I
- 19:30have a 300 page PDF in there, and
- 19:32let's say deep in that PDF, there are
- 19:34lots of tables and good like documentation stuff
- 19:37in it and like real statistics,
- 19:39Only the
- 19:40first, let's say, 25 pages are getting indexed
- 19:43in the semantic index. Only the first four
- 19:45megs, essentially, of a piece of content is
- 19:47getting indexed. Now, granted, if I got 300
- 19:49page PDF, usually only about 15% or 20%
- 19:52of that is actual content, and the rest
- 19:53of it is all like packaging
- 19:55and Adobe PDF crap and all that kind
- 19:57of stuff. Same thing with Word Docs and
- 19:58PowerPoints. But a lot of people look at
- 20:00this, they say, I know the answer is
- 20:02in that PDF. And let's just say it's
- 20:04on page two seventy four. But when I
- 20:07ask Copilot, it doesn't know the answer. It's
- 20:08because the semantic index doesn't get the entire
- 20:11file. It's limited. And it's just part because
- 20:14we're early with all this AI stuff, and
- 20:16Microsoft is still working to get the things
- 20:18big to make the the the capabilities of
- 20:20what it can put in the index much
- 20:22bigger. So I I think it's big today.
- 20:24You you're already at 512
- 20:26megabytes today. That four meg has been surpassed.
- 20:29So if that goes even higher, like, I
- 20:31don't know. I think about the majority
- 20:33of organizational
- 20:34content that you work with is not many
- 20:37of us are working with 500 meg word
- 20:39documents, maybe Excel sheets, thing things like that.
- 20:42But that that number always gets bigger and
- 20:44bigger, probably up to the limitation of just
- 20:46file size in SharePoint at some point.
- 20:48But context windows need to grow. I I
- 20:51totally agree. I like context there. I actually
- 20:53have a one of my coaching clients is,
- 20:55in the, let's just say, the national security
- 20:58industry in The United States, and they're involved
- 21:00with the Department of Defense, and
- 21:03let's just say nuclear is involved in it
- 21:04as well. And they have some PDFs that
- 21:07are like three and four gigs of, like,
- 21:09nuclear safety stuff, and they're like, Why are
- 21:11we not seeing this stuff? So what we
- 21:13had to do instead to fix that was
- 21:15instead of just putting the stuff inside of
- 21:17a SharePoint document library, which they weren't doing
- 21:19anyway, but that's a whole head Snowden kind
- 21:21of like be careful.
- 21:23Instead, what they had to do is they
- 21:24had to create a custom process that would
- 21:27take all content out of that PDF.
- 21:29They would then create the embeddings
- 21:31and then store it in a vector database,
- 21:34and then
- 21:35they could go through and find all the
- 21:37content based on the user's prompt. Best way
- 21:39to do that, use Azure AI search. It's
- 21:41another resource. And you can hook that up
- 21:43to a Copilot based agent, or you can
- 21:45write a REST API that a declarative agent
- 21:48that a developer could write to go get
- 21:49that data, and now you have full visibility
- 21:52in the document. The big difference there, you
- 21:54as the developer, you have to pay for
- 21:56the Azure AI search
- 21:58resource and how much which is a a
- 21:59metered based thing. So how much you're putting
- 22:02in it and how much you're actually using
- 22:03it. Whereas, if you're doing a semantic index,
- 22:05that's part of the $30 per user per
- 22:07month all you can eat plan. Right? So
- 22:09Yeah. That's a there's a way around that.
- 22:11And, also, the Azure AI search doesn't have
- 22:13the security trimming because they don't have the
- 22:15capability to do ACLs
- 22:16on the stuff today. So that's a little
- 22:19Hey. It's it's a you can't have your
- 22:21cake. You need it too Right. Depending depending
- 22:23on where you are. I I think there
- 22:24is friction there. Like, one of the exciting
- 22:26things about context windows growing and effectively this
- 22:29concept of, like, quote, unquote, AI memory, right,
- 22:32and and chat memory and all these things
- 22:34is you're not gonna have to do that
- 22:36extra hop
- 22:38of potentially
- 22:39multiple chunks. Like, I I do run into
- 22:41customers in this situation today to really massive
- 22:44PDFs, things like that, where they have to
- 22:46pre chunk, and then they have to go
- 22:48and vectorize that to generate they have to
- 22:51go generate embeddings, store all that stuff in
- 22:53a vector database, be it Pinecone, AI
- 22:56search, whatever it happens to be,
- 22:58and manage it through. But if you've got
- 23:00some maturity and, like, you're there and you
- 23:02understand that stuff and you're kind of on
- 23:04the, quote, unquote, cutting edge, like, sure. It's
- 23:07right there.
- 23:08It it is available for you. I've always
- 23:11been surprised, like, even watching, like, PMs on
- 23:13our team be able to pick some of
- 23:15that stuff up and and just run with
- 23:17it, like, out of the box. Well, not
- 23:19out of the box. Like but the the
- 23:21number of us now that are sitting here
- 23:23writing Python scripts on the side to do
- 23:25weird machinations, like, it it it's ever growing
- 23:29as as as we get up here and
- 23:30and we do these things.
- 23:32We we've been we've been prepping for our
- 23:33build session all week and thinking about, like,
- 23:35oh, which model do we show? How are
- 23:37we gonna demonstrate checkpointing here?
- 23:39What what are we gonna do in this
- 23:41scenario, in this thing to to really show
- 23:43it off where it shines? I'm not at
- 23:45the point of writing my own Python scripts
- 23:46yet. I am at the point of asking
- 23:47chat gbt telling it what I want, and
- 23:49it writes the Python script for me. But
- 23:50I'm not I'm not the point of writing
- 23:51my own. Yeah. Well, nobody should be writing
- 23:54Python from scratch anyway these days. Let let
- 23:56me yeah. I do it for you. Yeah.
- 23:58Those were the really big things that I
- 23:59saw, and I know they're not exactly dev
- 24:01stuff, but I I don't know what else
- 24:03you guys wanna talk about in terms of,
- 24:04like, dev specific things. So I have another
- 24:06question that I think you said this in
- 24:09the video. I think we're okay to talk
- 24:11about this.
- 24:12Well, it's something that that This is the
- 24:14last podcast I'll have my my my MVP
- 24:16NDA. Well, that's why we can edit. Right?
- 24:18Todd's the only one listening. Todd, if you
- 24:20hear this is NDA NDA or anything about
- 24:22this is NDA, don't I know Todd. Todd
- 24:24will be good. Like, we talked about SharePoint
- 24:27index, surfacing content, size of files. One of
- 24:29the things you used to do or you
- 24:31still do to help with SharePoint search and
- 24:34digging through libraries and lists and all that
- 24:36is metadata on these files
- 24:37to be able
- 24:39to do, like, filters on it and query
- 24:41on it. And then you get a query
- 24:42and you get your filter down the left
- 24:43side with your metadata and doing all of
- 24:45that. How does Copilot handle
- 24:47metadata
- 24:49and files? That's a great question. It's a
- 24:50great answer.
- 24:52It doesn't.
- 24:53So here's the thing. Copilot only okay. So
- 24:56it's not so much Copilot. Remember, all goes
- 24:58back to semantic Right. Semantic index. The semantic
- 25:01index only has the content from files inside
- 25:04of SharePoint and OneDrive. It does not have
- 25:07content related to the metadata on those files
- 25:10in a document library. It does not have
- 25:12data from SharePoint lists. It does not have
- 25:14data from SharePoint pages.
- 25:16Microsoft knows all of this. They that is
- 25:18might even has changed by the time I'm
- 25:20saying this. It might even change by the
- 25:22time you're hearing this, but especially because build
- 25:24is next week because it's a huge ask
- 25:26for people. But I know, like, list data
- 25:28has been a huge blind spot when it
- 25:30comes to m three sixty five Copilot because
- 25:32it is that content is not being indexed
- 25:35into the semantic index. Only the files inside
- 25:38of a document library in OneDrive are being
- 25:40indexed. Okay. And not even so not even
- 25:42ASPX pages because technically, I think of ASPX
- 25:44pages and, yeah, you look at companies' intranets
- 25:47or I know companies,
- 25:49good idea or not, have built, like, whole
- 25:51wikis and all their documentation in ASPX pages
- 25:54because they want it in that format.
- 25:56And then so they could also be very
- 25:59surprised when they go ask Copilot, and Copilot's,
- 26:01like, missing data. Maybe it's missing data from
- 26:04Power Apps that are writing to SharePoint lists
- 26:06or Yeah. You're a developer or SPFX. I
- 26:09can do SPFX
- 26:10writing the SharePoint list. None of that's gonna
- 26:12come into
- 26:13Copilot.
- 26:14That's correct. But, again, I just wanna stress
- 26:16one thing, and that is that is the
- 26:18way it at least was
- 26:21recently.
- 26:21And I would whenever something like this comes
- 26:24up and cope with with, like, a limitation,
- 26:25it's always, I'm filing this back in my
- 26:27head. When I do a search, something's not
- 26:29happening. It's like, I remember that guy told
- 26:31me this thing at one point. Yeah. I
- 26:33wonder if that's going on. Let me go
- 26:34research or go to your research and be
- 26:36like, well, no. Andrew's wrong. I just saw
- 26:38this build this week, and they said that
- 26:41now we're doing list content, like, going, yeah,
- 26:42well, that's probably that's a couple days after
- 26:44we recorded this. So
- 26:45this is changing. Microsoft is is like they're
- 26:48feverishly working Right. On everything with this to
- 26:51make it more powerful. So this stuff could
- 26:54is I would expect it to change and
- 26:56get better, but it's just one of the
- 26:57things you wanna be, like, let's double check
- 26:59and make sure this is the case. That's
- 27:00okay. Right. That's turned into my default answer.
- 27:02And I'm doing demos now or something, and
- 27:04I can't find something. I'm like, must have
- 27:06changed because it does. It feels like this
- 27:08stuff is changing, like, daily or hourly or
- 27:12by the minute.
- 27:13I was teaching a class this week. I
- 27:14recorded a demo at 10:30 at night. I
- 27:16taught the class at nine the next morning,
- 27:17and it literally was different at nine the
- 27:19next morning in the exact same tenet. And
- 27:21so I was like, Yeah, stuff is flighting
- 27:23for build, so my bad. Welcome to Evergreen
- 27:26SaaS products.
- 27:27Exactly. Right. And this time of year is
- 27:29always fun too because the documentation
- 27:32often freezes before these events. So sometimes you
- 27:34start seeing new things come out before
- 27:37documentation's even there or or anything else along
- 27:40the way. So it's it's a little bit
- 27:41of a
- 27:42a challenge to to keep up with and
- 27:44balance. But I think, like, we've been talking
- 27:46about this for years and years and years
- 27:47with just Office March, m three sixty five
- 27:50in general. Like, the only constant is change
- 27:52there anyway. So you gotta kinda be ready
- 27:55ready to pivot and and deal with those
- 27:57things
- 27:58as they do come up. And I think,
- 27:59like, back to earlier about those customers who
- 28:02are mature enough maybe
- 28:04to go ahead
- 28:06and chunk and and generate embeddings for for
- 28:09their own stuff,
- 28:10they also know that they're kind of on
- 28:12the bleeding edge and need to keep up
- 28:14with some of this stuff.
- 28:16Yep. I I do worry sometimes about the
- 28:19not that's not like some, like, heavy hearted,
- 28:21like, the world's gonna end kind of thing,
- 28:22but just about the friction for customers there,
- 28:24just as the buttons are moving every day,
- 28:26the functionality is changing. Like,
- 28:29oh, I I I can do
- 28:31deep research today in in Copilot consumer, but
- 28:34I can't do it yet in m three
- 28:36sixty five Copilot. Like, those disparate experience experiences
- 28:40and things, they they do tend to
- 28:43add up. So you still gotta do the
- 28:44TCO and figure out, like, where value is
- 28:47inflicted in my org.
- 28:48I completely I completely agree with that. I
- 28:50mean, it's like it's one of the things
- 28:52that I I find is like a service
- 28:54offer for my customers. I I I do,
- 28:55like, this coaching service, and it's like, hey.
- 28:57Look. You can't stay on top of this
- 28:59stuff. I'm the one that's my job. I'm
- 29:00tango staying on top of it. So it's
- 29:02pretty much we jump on a call. It's,
- 29:03like, going, what do you know that's different
- 29:04based on what we're doing? And I'm, like,
- 29:05going, okay, this is no longer the case,
- 29:07or this is the case, or be careful
- 29:08with this, or I know that this is
- 29:09gonna change, so I would not invest time
- 29:11here because stuff is gonna change in the
- 29:13next few months. So it's it's just the
- 29:14only thing that's different about what you said
- 29:16to me, Scott, is that it's the change
- 29:18is just going so much faster. Like, we
- 29:19are on a much steeper downhill than we
- 29:21were on the mountain before. For sure. I
- 29:23I think pace to change is is very
- 29:25real there. So should you get into this
- 29:28world, all of a sudden, you end up
- 29:30in a place where
- 29:32you're kind of joining the space race of
- 29:33everybody else. And and you gotta be prepared
- 29:36for that. So the solution that you build
- 29:39today, should you take a dependency
- 29:41on any kind of
- 29:43SaaS surface or anything like that, it's or
- 29:45or a PaaS service even, you have to
- 29:47know that it's gonna change. And it's probably
- 29:49gonna change more rapidly than you're used to
- 29:52in the past. That said, if you're still
- 29:54doing some of this stuff maybe in, like,
- 29:56an on prem context, like, it's one thing
- 29:58to say, hey. I'm using Azure AI studio
- 30:00or Azure AI search, and it's moving at
- 30:02this pace. Or
- 30:04I'm using AI Foundry, and it's going this
- 30:06fast versus
- 30:07I I took a dependency
- 30:09on langchain and Pinecone here, and and that
- 30:11was locally for me to do. So you
- 30:13you can have a little bit of that
- 30:15stability along the way if you need it.
- 30:17Yep. It's not gonna be super stable. Like
- 30:19you said, the hill is still a hill.
- 30:21And and it's and it's a very it's
- 30:22it's a very steep one. Yeah. That we're
- 30:25all falling down or trying to climb up,
- 30:26I can't figure it out.
- 30:28It's I think it's a little bit of
- 30:29both. I can I can make a little
- 30:30bit of progress, but I fall four rings
- 30:32down?
- 30:33Yes. Yeah. %. So I know, Scott, you
- 30:36have meetings coming up. I have meetings coming
- 30:39up. AC has
- 30:40coaching and meetings coming
- 30:43up. But anything else, AC, any last thoughts,
- 30:46other things people should be aware of or
- 30:48even if people wanna learn more about Copilot
- 30:50dev? I know we didn't get into all
- 30:52the developing your own agents. Maybe we'll have
- 30:54to have you on for the fifth or
- 30:56sixth time or whatever and talk more about
- 30:58the custom dev story, but, yeah, people wanna
- 31:00get in touch with you, any of that.
- 31:02Yeah. I I appreciate that. I guess it
- 31:04I would love to come back on happy
- 31:06to talk about the dev story. In fact,
- 31:07it'd probably be better if we did it
- 31:08after build too so that I can Alright.
- 31:09I know some stuff that's coming that we'll
- 31:11talk about that Microsoft will talk about next
- 31:12week. But, yeah, I have a company called
- 31:15Voitanos. I focus on making you the best
- 31:18Microsoft three sixty five developer in the Microsoft
- 31:20three sixty five ecosystem
- 31:22and indispensable in your organization. So I do
- 31:24that through, like, SharePoint framework training, Teams app
- 31:27dev training, and Copilot training. I have a
- 31:30a six hour,
- 31:31two day virtual workshop that I do on
- 31:33building declarative agents for the developer audience, so
- 31:36not using Copilot Studio, but using Visual Studio
- 31:39Code. And I'm due to refresh that. I'm
- 31:41intentionally waiting until after build because I know
- 31:44things are gonna be changing. But I'm gonna
- 31:45I plan on I do that at conferences
- 31:47for a full day workshop. I did it
- 31:48at the Microsoft three sixty five community conference.
- 31:50I'm gonna do it again at the TechComm
- 31:51conference in Seattle in June. But I also
- 31:54have that as a, as a live workshop
- 31:56that I'll offer on my site at voitonos.io.
- 31:58And if you, if you if you're interested
- 32:01too, I mean, I've got, like, the recordings
- 32:02from when I last delivered that in March
- 32:04are also available
- 32:05for, to enroll in on my course where
- 32:07you'll actually get access to the the updated
- 32:09ones once those are done. So, yeah, I
- 32:11mean, the best place to find me is
- 32:12just voitanos.io
- 32:14or search for I I'll put give you
- 32:16guys my link so you can put them
- 32:17in the show notes, but Perfect. LinkedIn, Blue
- 32:19Sky, YouTube,
- 32:20I'm easy to find. If you just search
- 32:21for Andrew Connell in SharePoint, it's I'm pretty
- 32:24much the first one that's gonna come up.
- 32:25Perfect. Sounds good. Well, thanks, AC. Thanks for
- 32:28jumping on with us, filling in some of
- 32:30those additional details around CoPilot agents. And, appear
- 32:34I was gonna say the message takeaway from
- 32:36this is go pay attention at build, but
- 32:39this is gonna come out after build. So
- 32:41if you're listening to this Go back and
- 32:42watch build. Go back and watch build and
- 32:44look for maybe certain announcements around Copilot agents
- 32:48and some of the stuff you may have
- 32:49heard here. Watch for my recap. I will
- 32:51have a recap for eight when it comes
- 32:53to SharePoint, Teams app dev, and agents that
- 32:55is, like, the build stuff, like, this is
- 32:57what you wanna focus on. I did it
- 32:58for Ignite. I did it for Build. It's
- 33:00a great way for me to get get
- 33:02up to speed on everything, and I plan
- 33:03on doing it again as Build goes on
- 33:05next week as well. Okay. Perfect. And if
- 33:07you have that
- 33:08before this show goes live, send it over,
- 33:11and we'll include that in the show notes
- 33:12as well for anybody listening. If you wanna
- 33:14get AC's recap, and if it's live or
- 33:16going to be live, we'll add that as
- 33:18well. Awesome. Sounds good, man. Awesome. Well, thanks,
- 33:21guys. Appreciate it. Yeah. Enjoy your weekend, and
- 33:23we'll talk to you guys later. Thanks. Alright.
- 33:24Thanks, Ben, Scott. And congrats again on episode
- 33:26400. That's awesome. Alright. Thanks, AC. Appreciate it.
- 33:29Bye.
- 33:31If you enjoyed the podcast, go leave us
- 33:34a five star rating in iTunes. It helps
- 33:36to get the word out so more IT
- 33:38pros can learn about Office three sixty five
- 33:40and Azure.
- 33:41If you have any questions you want us
- 33:43to address on the show or feedback about
- 33:45the show, feel free to reach out via
- 33:47our website, Twitter, or Facebook.
- 33:50Thanks again for listening, and have a great
- 33:51day.