Latest / Microsoft Cloud IT Pro Podcast / Episode 325 – How do I migrate from SharePoint Online to Azure blob storage?
Transcript
- 0:03Welcome to episode 325 of the Microsoft Cloud IT
- 0:08Pro Podcast recorded live on March 10th, 2023.
- 0:12This is a show about Microsoft 365 and Azure from the perspective of it pros and
- 0:17end users where we discuss a topic or recent news and how it relates to you.
- 0:22Microsoft Loop is finally coming in public preview later this month.
- 0:26So naturally,
- 0:28Ben and Scott share some of their experiences with Loop and in particular loop
- 0:32components to date as well as a few oddities that they've run across using Loop
- 0:36components. From there,
- 0:38they move into an interesting scenario they recently encountered around moving a
- 0:42hundred plus terabytes of data out of SharePoint and into Azure blob storage
- 0:47and some of the challenges that one might encounter in such a scenario.
- 0:53I have a question, Scott. When can I start playing with Loop ?
- 0:58I don't know.
- 0:59Off the top I can say that Loop is a confounding
- 1:04product to me. The more you make me interact with it,
- 1:08the more confused I get by it. So example,
- 1:13like real practical example,
- 1:15the other day we were recording the stream for YouTube and we're putting that
- 1:20together.
- 1:21And normally when we do the YouTube streams we go back and forth inside
- 1:26of just teams chat with, with random ideas and kind of, you know,
- 1:30status if we see somebody's dropped out or you know,
- 1:32their video froze or something like that.
- 1:35And there was this need to put in just a bulleted list for things.
- 1:40And it just so happened that in the tendency that you and I do,
- 1:43teams in like Loop is all lit up and ready to go. So as a, you know,
- 1:48great smart ass, I went in and said, Ooh,
- 1:51rather than creating some rich text here,
- 1:53I'm gonna create a loop component cuz it's gonna be great.
- 1:55So the scenario is you and I are in a teams chat,
- 1:58we're in the same exact teams chat. We both have permission to the chat,
- 2:01we're like in a live meeting, right? We're,
- 2:03we're going through and we're doing it. I wanna.
- 2:05Put this on top of it. We're in, we're logged into the same tenant too. Yes.
- 2:08Like we're both using IDs in the same tenant.
- 2:10This isn't external guest user or uh,
- 2:14I invited an external user to the meeting.
- 2:17This is you and I logged into teams into the same tenant in a chat in a meeting.
- 2:22Yep. So I went two of us in a chat in a scheduled meeting.
- 2:26This isn't even like a one-off teams chat or anything like that.
- 2:29I don't know if that would make a difference, but just to like,
- 2:32this is a fairly structured thing and,
- 2:34and kind of a common workflow I would think.
- 2:37So we're in the chat and I create a loop component which
- 2:41is just rich text and it's got a couple bullet points in it and I write up the
- 2:46text that I'm gonna put in there and I hit send the send button in Teams
- 2:51first problem.
- 2:52The loop component does not render for you at all and you can't even see it
- 2:57rendering there.
- 2:58You get basically a link to click on and go out and view it online
- 3:03at which point you get an access denied message.
- 3:07So you have to request permissions to the loop component that I just
- 3:12sent to you in that teams chat,
- 3:14which in and of itself I'm just gonna say like that is a horrible
- 3:19experience. Like yeah,
- 3:20teams should know that you and I are chatting with each other and that I
- 3:25explicitly created that.
- 3:27I didn't send you a preexisting loop component so it wasn't like I was sharing a
- 3:30document or a random link that you, you shouldn't have had access to.
- 3:33It's like I created this loop component as a part of this chat.
- 3:37So you had to go open up, you know,
- 3:40the web view of my OneDrive request access.
- 3:43I had to go and know where to go in my OneDrive to respond to that access
- 3:48request or wait for the email to come through from SharePoint that says,
- 3:52oh you know Ben requested access to your thing. I give you access.
- 3:57I have to tell you in real time, mind you,
- 4:00this is like minutes later as we've been dancing around and trying to figure out
- 4:03what's going on here,
- 4:04I have to tell you that I gave you access to that component. And even then,
- 4:08because teams is relatively static when it comes to chat and things like that,
- 4:13it's not like auto refreshing in the background for messages that were
- 4:17previously sent.
- 4:19Then you have to click out of the chat in the meeting and come back in,
- 4:23which I'm gonna argue doesn't really work like it works if you're somebody who
- 4:29always chats outside of your team window.
- 4:32Like you and I are in a meeting right now and we're doing video back and forth.
- 4:36So I tend to chat out of the same meeting, uh,
- 4:39the same window that the video's in and the only way to refresh that would be to
- 4:44leave the meeting and then come back like what a horrible
- 4:49broken, awkward experience.
- 4:52Like it really makes me question if the people who want us to use loop
- 4:57components have ever tried to use them themselves.
- 5:02This is a little odd to me.
- 5:03So I have seen this done and I thought I had done this
- 5:09successfully in the past and I've seen people do this successfully
- 5:14in the past where they've all been in a meeting or in a chat and they've gone
- 5:18and added a loop component and everybody's been able to access this.
- 5:22So I'm also a little curious is, is this,
- 5:26do I have a setting in my tenant that I missed? Is this a timing thing?
- 5:31Is this like why is it doing this? Because I have, in all fairness,
- 5:36I have seen this work, but to your point,
- 5:39it shouldn't be this complicated is to why it doesn't work. Like.
- 5:43I'm gonna argue if there's a setting in your tendency that you need to set to
- 5:46make that work, it shouldn't, again,
- 5:48that should not even be a consideration if there's something
- 5:53out there that you need to go ahead and crank on a knob like that.
- 5:58Sorry, like that's just a,
- 6:00a non-starter And I might be a little salty about it cuz we've been doing like
- 6:05planning and a bunch of like product wor work this week and I've been spending a
- 6:08lot of time with customers and thinking about customer asks and things
- 6:11like this. This is just, I, it's one of those ones where I go,
- 6:17did anybody ever try this?
- 6:19I wonder, I just tested this. So I put a link in,
- 6:22I put a screenshot in the discord of what this looks like and it showed a loop
- 6:26list in draft and it gives me like a message at the top.
- 6:29Some recipients won't be able to view or edit this and this was in a chat with
- 6:34me and someone else in my tenant. I didn't actually send this one to you Scott,
- 6:38it was with somebody else.
- 6:40And I waited a little bit and I finally clicked send and then I got a little
- 6:43button that said refresh. And now that I refreshed it and I look at access,
- 6:49it looks like they have access. I'm speculating.
- 6:54I wonder if there's a timing thing because it did just work for me,
- 6:59but it was not as quick as you and I did it before,
- 7:03like it was seconds while you were talking where I started it up and maybe
- 7:07waited 20,
- 7:0830 sec 20 or 30 seconds or so and it did work.
- 7:13Like is it just slow in going in and granting access to it?
- 7:17Where had we taken longer? Again, to your point,
- 7:21I would argue it should probably be quicker and a little bit more
- 7:23straightforward of ex explaining why it didn't work,
- 7:27but it did just work for me. But it was slow. We.
- 7:30Waited what like minutes the other day cuz we did a whole big dance back and
- 7:33forth of going out to OneDrive and you requesting permissions.
- 7:37So I'm gonna put on my hey I ideation hat thing, right? Yeah,
- 7:42like how, how should that work?
- 7:43So your argument that it might take a little bit for permissions to propagate,
- 7:48you should tell me that in the client.
- 7:49Like you shouldn't even display the component,
- 7:52put a spinner over for it or something and say like working on it and you need
- 7:56to wait like asynchronously until the other end gives an act and says,
- 8:01Hey, all the permissions have propagated and we're ready to go.
- 8:05If there's some type of weird asynchronous flow that's going on where like when
- 8:10you click a send button,
- 8:11it's creating the loop component and then you're waiting on another async call
- 8:15to do the permissions later. Uhhuh .
- 8:17Like there could be a weird timing issue there.
- 8:19Like don't make that an asynchronous call, make that a synchronous thing.
- 8:23Create loop component set permissions display, right?
- 8:27Like and then display block. Yeah, just block the display of it.
- 8:30don't make me have the cognitive load and
- 8:35the overhead to think about that. Especially in a world, uh,
- 8:39where you and I are in the same exact tenancy, same a d tenant,
- 8:44same set of permissions, like all the, like all those things.
- 8:47You're not logged into that tenant right now, are you?
- 8:49Cuz I can't even send you one in the call we're in now.
- 8:53Probably not. I.
- 8:53Wanna go back and try this again anyways.
- 8:55We can go play with it more. But I think yes, it was one of my,
- 8:59that's only like the second or third time I've interacted with Loop and
- 9:04it has been a showstopper frustrating experience each
- 9:09and every time I've done it that it makes me question
- 9:14sanity as a user,
- 9:15which isn't a good place for any product to be. Yes.
- 9:21I would agree with that, but Scott,
- 9:23you can play with it more now because we've had loop components.
- 9:28But I did see that the Microsoft Loop app,
- 9:31so now we have a standalone web app for Loop is coming to public preview
- 9:36this month is the current plan. It's gonna start rolling out over this month.
- 9:41Go check your message center because there's a whole bunch of stuff in here.
- 9:44I actually saw it on Twitter before I saw it in the message center and
- 9:49they haven't figured out here. Here's a shocker for you Scott ,
- 9:52they haven't figured out the compliance stuff for it yet.
- 9:57So while it's coming in late March,
- 10:01it is not enabled by default coming late March.
- 10:06They are rolling it out,
- 10:08but it is going to be disabled in tenants by default and you actually need
- 10:13to go over to your cloud policy service. So the,
- 10:17what is it? config.office.com. Yes.
- 10:22config.office.com and go tweak your policies and
- 10:26config.office.com to enable the Loop app so that when it does come,
- 10:31so you can go enable it now, like it's in the policy.
- 10:34I already went and enabled it so that when it does get deployed to your tenant,
- 10:38you will be able to use it and play with Loop even more because I know that's
- 10:41what you wanted to do. Scott .
- 10:43I really don't know that I want to, it's .
- 10:47It's just another, it's another one of those things. I think it's going to be a,
- 10:53it's gonna be a very SharePointy experience and I often feel the rough
- 10:58edges on SharePointy experiences. Like I always want them to do more.
- 11:02And I think we talked about this the other the other day too, but uh, we,
- 11:06we did it a little bit more privately. We can do it again here. Uh,
- 11:10Purpose-built tools are purpose-built for a reason.
- 11:15Like they're very good at doing that one thing.
- 11:18So if I look at say, component or a piece of SharePoint,
- 11:22like lists, lists is like, okay,
- 11:25but it carries just all this overhead and baggage of being ultimately
- 11:30just a SharePoint list, right? It's not really a Microsoft list,
- 11:33it's a SharePoint list and Carries, that's a SharePoint list,
- 11:35all the constraints that come with that. So rebranding aside,
- 11:39like it's got a lot of cruft and it's got a lot of overhead and it's got
- 11:44some weirdness to it because it was designed to live in SharePoint.
- 11:48I fundamentally believe that, right? Like there,
- 11:50there's constraints in the underlying platform that drive to, you know,
- 11:54how these things come out and, and are manifested for end users.
- 11:58So if I think about like Microsoft lists and then Uhhuh ,
- 12:01maybe you go use Airtable, sorry,
- 12:04air tables a hundred million times better like . It, it is,
- 12:09but it's purpose built.
- 12:10Do I get the document integration and Power Automate and all those things?
- 12:14Like absolutely not. Like I don't get that whole ecosystem,
- 12:18but if I don't need that ecosystem,
- 12:21like a purpose-built tool might actually work a little bit better for me.
- 12:25And the reason this is kind of coming to mind for me is that lots of people
- 12:30compare Loop and especially kind of the standalone loop, like,
- 12:35hey, let's go to loop.microsoft.com as a notion replacement and
- 12:40notion is for all its flexibility,
- 12:44a very purpose built tool.
- 12:46Like it does have a line where it stops trying to be the everything for
- 12:51everyone kind of tool and it knows its limitations and it just,
- 12:56it wears those words right out there.
- 12:58Like you can find it in the documentation and you just know you go like,
- 13:02oh, this isn't gonna work.
- 13:04Versus something in SharePoint land and Microsoft 365 Land,
- 13:09quite often the answer is,
- 13:10well like that could work if you did this and this and maybe it'll work if you
- 13:14did this and what if you integrated this thing over here and you did this and
- 13:17you're like, I I I don't want to do that.
- 13:20I just kind of like want to to work and I want to do work,
- 13:24so enable me to do some work. Please.
- 13:26Yeah, and I think I totally agree.
- 13:30Like I think Loop could have been or could be,
- 13:33it'll be interesting to see when it comes out and we continue to play with it
- 13:36more. But in your,
- 13:38to your point like was SharePoint and OneDrive fundamentally the
- 13:43right place to store all of these
- 13:48files that are going to be your loop components, your loop pages, all of that.
- 13:52Ultimately those are gonna live in SharePoint. Just like you said,
- 13:56Microsoft lists are SharePoint lists, OneDrive document libraries,
- 13:59they are SharePoint document libraries.
- 14:02And there are absolutely fundamental things that,
- 14:08to be fair,
- 14:08technical deck carried all the way through from the early days of
- 14:13SharePoint. For those of you that have dealt with lists,
- 14:16maybe you haven't dealt with SharePoint lists and specifically,
- 14:18I still run into this all the time,
- 14:20is the whole 5,000 items in a query and in a list is like,
- 14:24I don't know if it was 2003, 2007,
- 14:27but fundamentally it's because all of this stuff lived in a sequel database.
- 14:30When you would go to query more than 5,000 items from the sequel database and
- 14:34would put locks on the database,
- 14:35that would cause the entire platform to run slow carry it forward
- 14:4015 years. And we're still dealing with it because of that technical debt.
- 14:44And when you do things like Loop and SharePoint lists and you try to treat a
- 14:49SharePoint list like a database, you inherently run into these things.
- 14:53To your point, other places are much better to build our li large lists
- 14:59because that's what they were designed for.
- 15:01And when you try to make a SharePoint list a database or even this
- 15:06like are we trying to make SharePoint the storage space for Loop when it should
- 15:11be like, oh,
- 15:12living in blob storage or living in some other backend service
- 15:17that isn't going to introduce some of the constraints.
- 15:22Now I also get the fact that SharePoint makes sense because you already have all
- 15:26the identity stuff, the permissioning stuff, the sharing stuff.
- 15:29There's a lot of stuff they don't have to build by putting it in SharePoint,
- 15:34but they also do incur that technical debt and those technical limitations
- 15:39that live in SharePoint really because of how it was built 20 years ago. Yeah.
- 15:45.
- 15:45It's interesting.
- 15:50Do you feel overwhelmed by trying to manage your Office 365 environment?
- 15:54Are you facing unexpected issues that disrupt your company's productivity?
- 15:57Intelligent is here to help much like you take your car to the mechanic that has
- 16:01specialized knowledge on how to best keep your car running Intelligent helps you
- 16:06with your Microsoft Cloud environment because that's their expertise.
- 16:09Intelligent keeps up with the latest updates on the Microsoft Cloud to help keep
- 16:13your business running smoothly and ahead of the curve.
- 16:16Whether you are a small organization with just a few users up to an organization
- 16:20of several thousand employees,
- 16:22they want to partner with you to implement and administer your Microsoft Cloud
- 16:26technology, visit them at intelligent.com/podcast.
- 16:31That's I N T E L L I G I N
- 16:36k.com/podcast for more information or to schedule a 30 minute
- 16:40call to get started with them today.
- 16:43Remember intelligent focuses on the Microsoft cloud so you can focus on your
- 16:48business.
- 16:50You also experienced a question going back to using things for what they're
- 16:54built for and challenges that can come with it.
- 16:57What about if you try to copy stuff out of SharePoint with AZ copy?
- 17:01I had a question from a colleague come in and I think it's an interesting
- 17:05scenario. Uh, so first let's kind of start with the question.
- 17:10I have a colleague that's working with a customer and that
- 17:15customer is looking to migrate a couple terabytes of data
- 17:20from SharePoint online to Azure blob storage.
- 17:24Like they clearly wanna pull some of those files, documents,
- 17:28whatever they happen to be out of SharePoint and push them over to Blob
- 17:33in and of itself. Like, okay, fine.
- 17:35There were some things that didn't even cross my mind in this
- 17:39conversation when I was chatting with a person. So the initial ask,
- 17:42like just to the number that they asked for was higher than this. Well,
- 17:45like let's peg a number to it, right? Like,
- 17:47so I wanna migrate a hundred terabytes of data,
- 17:50which means I have a tendency with approximately 10,000 users give or take
- 17:56to get like the requisite licensing, right?
- 17:58It would be someplace in that order of magnitude, right? You,
- 18:00you're more than five licensed territory or, or you've bought a,
- 18:03a whole big chunk of storage from Microsoft,
- 18:06but they wanna migrate a a hundred terabytes of data from SharePoint online
- 18:12to Azure Blob and they're already trying to do it. And the,
- 18:15and the way they're trying to do it is using a storage explorer,
- 18:20which I have to imagine, like, I didn't even ask the question,
- 18:23but I have to imagine what they're doing is either using synced libraries that
- 18:28are already synced through OneDrive for business or they're using
- 18:33the, like the file explorer view and kind of the mm, the,
- 18:38the wonkiness that comes along with like that,
- 18:41that weird web a not web d a thing that they have going on these days.
- 18:45That thing that should have never been in SharePoint to begin with.
- 18:48Never should have been around to start.
- 18:51I I guess a couple of things here. So one is, uh,
- 18:55storage Explorer as a tool uses AZ copy under
- 19:00the hood for its underlying copy operations.
- 19:03So you can think about like dragging and dropping a file from
- 19:08say,
- 19:08your desktop into a storage explorer window that's mapped to a storage account
- 19:13in Azure as an operation where Storage Explorer is acting
- 19:18as a nice wrapper and a nice proxy and it's gonna build out and it's gonna say,
- 19:23Ooh, I had a file dropped in here,
- 19:24so I need to call AZ copy CP with this source,
- 19:28this destination and push all those things through.
- 19:31And if you actually look in the, the logs for Storage Explorer,
- 19:34you can see that it's calling the commands for AZ copy in the background for
- 19:38those kind of core copy operation things, right? The the puts and the GIS and,
- 19:42and all that kind of stuff. So.
- 19:43Just quick question,
- 19:44is there any performance loss by using storage copy versus just AZ copy in the
- 19:49command line?
- 19:49Or is it really gonna be about the same where Storage Explorer doing that
- 19:54type of stuff as just a friendly cover on top of AZ copy,
- 19:58kinda like the Azure portal is a friendly cover on top of rest calls?
- 20:01Depends on the operations you're doing. Like there's, okay,
- 20:04there are things that Storage Explorer does that are not functionality that's
- 20:08provided by AZ copy really we're talking about like the underlying data movement
- 20:12engine of storage Explorer.
- 20:13Is just AZ copy is.
- 20:15AZ copy and a standalone, uh, thing for it.
- 20:17So there might be instances where Storage Explorer is great for you,
- 20:21like you need a gooey, you need to browse it,
- 20:22you wanna do something like calculate folder statistics, anything like that.
- 20:26Like awesome, go ahead. You wanna view apples on like ADLs,
- 20:30gen two storage account ton of file or folder like great use storage explorer
- 20:34for that. Like that's native functionality to it. And then, you know,
- 20:37you wanna copy data, use AZ copy if you have a large data copy to do,
- 20:42you might not want to use Storage Explorer as an extraction layer because it's
- 20:47just adding more overhead at that point, right? It's,
- 20:49it's an orchestrator for you,
- 20:51but ultimately it's orchestrating a single instance of AZ copy.
- 20:55So if you're trying to do, you know, a large copy operation,
- 20:58and I'm gonna define large in like the hundred million plus file range,
- 21:03like say you're trying to copy half a billion objects, 500 million of those,
- 21:07you might wanna orchestrate that with like multiple running versions of AZ copy
- 21:12or even AZ copy running for multiple machines or things like that.
- 21:15So Storage Explorer may or may not cut it for you in that scenario, you know,
- 21:20as,
- 21:20as as kind of an orchestration or or or proxy thing that's sitting
- 21:25on top of, on top of easy copy. Got it. So back to our scenario.
- 21:29We've got a a hundred terabytes of data in SharePoint online,
- 21:33not very clear where it sits. Does it sit in, you know, a,
- 21:36a set of libraries in a single site? Is it multi-site, like whatever,
- 21:40I don't know that it matters, uh, a whole ton,
- 21:43but just from off the cuff from like the question I got, it's like, ooh,
- 21:46it sounds like they're maybe trying to do something unnatural here.
- 21:48So I was just trying to think through like what are their options And back to
- 21:53the, I don't always think about scale thing. I think the way you might,
- 21:58I saw a number that ended in TB and I was like, oh, it's terabytes.
- 22:02This is easy, right? Like you can totally use AZ copy,
- 22:05you can use AZ copy to do multi petabyte migrations. So like,
- 22:09we're good there we're golden. So my first thought was like,
- 22:12let's just sync it back to OneDrive for business.
- 22:16Like pull the data down locally in a client and then go ahead and continue to
- 22:21use Storage Explorer or just point AZ copy at that folder where OneDrive has
- 22:25synced locally and you know,
- 22:27don't do it on like OneDrive on like your client on your laptop while you're
- 22:30sitting on the beach.
- 22:31Spin up a VM in Azure that's in Microsoft's wan where it's gonna be able to do
- 22:36a semi performance synchronization through OneDrive for business.
- 22:41And then certainly like run easy copy from that op uh, from that VM as well.
- 22:45So you're all on the same backbone and,
- 22:48and everything's gonna kind of go a as quick as it can.
- 22:51And you quickly pointed out to me that there's some uh,
- 22:55some potential limitations just around OneDrive for business, whereas I'm like,
- 22:59I don't know, it's just a couple terabytes of data. Who cares? So.
- 23:02This was a really fascinating question.
- 23:04If there's a customer out there that's trying to do this and they want somebody
- 23:07to help them call me, cause I would love to do work through this,
- 23:10but this is not, again,
- 23:12from your perspective blob storage insignificant from a SharePoint perspective.
- 23:17So here's where my head instantly goes is continuing with our a
- 23:22hundred terabyte example, a hundred terabytes is one and a half.
- 23:26I just did the quick math 104 million megabytes
- 23:31SharePoint. Naturally,
- 23:33it's probably a lot of office files would be my assumption.
- 23:37I don't know for sure you'd have to go do some looking,
- 23:40but let's say your average file size is one meg,
- 23:42maybe even two megs PowerPoints.
- 23:45Excel could be bigger Word documents could be smaller.
- 23:48You're looking at like 52 million files where my head instantly went
- 23:53when you said sync it with OneDrive.
- 23:55Is OneDrive in the TSH or in the service description document limitations.
- 24:00And I've seen this, uh, play for optimum performance.
- 24:05Microsoft recommends storing no more than 300,000 files
- 24:09in a single OneDrive team or library site that you're going to sync.
- 24:14And if you are going to use sync,
- 24:17the same performance issue can be seen.
- 24:20If you have more than 3000,
- 24:23300,000 items across all libraries,
- 24:27you are syncing even if the libraries themselves are smaller.
- 24:31So if you're talking 52.5 million files and you can never
- 24:36sync more than 300,000 items,
- 24:39assuming 300,000 items are evenly dispersed across all your document libraries
- 24:44and sites and they're not all in one,
- 24:46it's a lot of syncing and unsyncing to get
- 24:51that amount of files onto a local computer where you could even begin to use
- 24:56something like AZ copy to copy them up. I mean maybe you go stand up, what,
- 25:00300,052 million divided by 300,000.
- 25:05You're doing it 175 times. No, maybe you're a large enough company.
- 25:10Again, a hundred terabytes,
- 25:12you're looking at probably 10,000 ish licenses,
- 25:17175 people to do this. If you can get a bunch of people working together,
- 25:22maybe not the end of the world,
- 25:23but now you're trying to coordinate who's thinking what, who's copying what,
- 25:27not missing anything. This is not an insignificant amount of work.
- 25:32Fair enough.
- 25:33So one of my other thoughts and I and I tend to go to this back from like my old
- 25:38migration days cuz I was used to always going from say like an on-prem
- 25:43SMB file share to SharePoint.
- 25:44You really didn't come back the other way too much, right?
- 25:46But you definitely had this machination, you went up where you,
- 25:49you went a certain direction and I think Microsoft has built out tooling over
- 25:52time like the SharePoint migration tool, uh, thing,
- 25:55things like that that can help you out. So my other thought was like,
- 25:58well hey let's go to our old friends at an ISV or like a migration vendor like
- 26:03sharegate and just point sharegate at that thing cuz it's got a script ability
- 26:07engine, it has PowerShell support, all those kinds of things.
- 26:10Mm-hmm .
- 26:10So let's say it wouldn't matter a hundred terabytes of data if it was one
- 26:15terabyte per SharePoint site.
- 26:16That's only a hundred sites of data you have to pull down,
- 26:20script it all out in something like a sharegate and let SHAREGATE
- 26:24just run again. Let it do its thing. So again,
- 26:27let's spin up a VM in Azure that's in the region probably with my storage
- 26:32account and just install sharegate on it, right?
- 26:35Point it at it and let it crank, let it download everything.
- 26:40Once it's all downloaded,
- 26:41then I'm just gonna push it up the other way with AZ copy and let it go.
- 26:45How viable is that?
- 26:46That's probably where I would go
- 26:51first.
- 26:54Although I was just looking so I couldn't remember. Cuz again,
- 26:57I've always gone the other way with you. Go get a share gate, go get a AvePoint,
- 27:01go get Metal Logics, which is now Quest I believe,
- 27:06which is actually now Dell, right? Dell owns Quest owns Metal Logics.
- 27:10It just keeps going. Yeah. So.
- 27:11Sharegate itself, I'd have to go look at Questa Metal Logics, sharegate,
- 27:15I head handy.
- 27:16Sharegate does not support a local file system as a
- 27:20Target. It supports it as a source.
- 27:25So you can go to your point from local drives up to
- 27:29SharePoint, but you can't go back down.
- 27:33If you're gonna go to a Target, it's SharePoint on prem online.
- 27:37One of the other third parties may do it Mover,
- 27:42which is now Microsoft Migration Manager maybe used to
- 27:46support it. They were pretty flexible.
- 27:48Though. I know. Uh, SPMT only goes one way. It's it's on-prem up kind of thing.
- 27:53Mover.
- 27:54Is too, cuz they've started pulling functionality out of the old mover.
- 27:58Do IO when moving it into Migration Manager,
- 28:00which migration manager is different than the SharePoint migration tool. Yes.
- 28:06for those who are listening in audio, like Ben's a big,
- 28:10my head just.
- 28:11Mind.
- 28:11Exploded, mind blown emoji right there. Yeah. I'm.
- 28:14Legitimately trying to think how would I do this?
- 28:17Like there's Logic Apps and Power Automate that let you Connect. That's.
- 28:22The next place I went to. I said, hey,
- 28:24if it's ongoing or even point in time migration,
- 28:27I don't know that it necessarily matters. Logic Apps,
- 28:30power Automate and Azure Data Factory have connectors.
- 28:35Or at the very least I can just do an HDP call and make everything work through
- 28:39the HDP connector and get it to where it needs to be.
- 28:43You might have some scaling issues in there given number of calls,
- 28:46number of API calls, things like that that you have to consider along the way.
- 28:51But it should be doable.
- 28:53Should be maybe. So I tried to do this in essence,
- 28:58I have a client that we're doing, we're not doing this,
- 29:02I was going to SharePoint from SharePoint, right or wrong,
- 29:06as this may sound.
- 29:07Their process and the way they did their work was they
- 29:12had certain projects, these were large projects, think like large,
- 29:17relatively speaking,
- 29:18not large compared to what you're used to storing large projects of a folder in
- 29:23SharePoint that would contain four or five, six gigabytes of data,
- 29:28uh, thousands of files.
- 29:30So much smaller than millions of files in terabytes of data.
- 29:34And their workflow was, once this project reached a certain point,
- 29:38it moved from one SharePoint site to another SharePoint site and then it had
- 29:42another status and moved again.
- 29:44And then eventually the part project got archived off and it moved again.
- 29:47So we did,
- 29:48I tried to do this with Flow originally where it was doing something
- 29:52similar to that where they could go flag a project, flag a folder,
- 29:57update metadata,
- 29:59whatever it may be to get flow to actually do the move in the copy.
- 30:03It didn't work. The data was too big.
- 30:07Ultimately what we ended up doing is I have a hybrid runbook
- 30:12in Power Automate that actually uses sharegate on a server sitting in
- 30:17Azure. So they trigger a Flow logic app.
- 30:22The Flow Logic app runs,
- 30:23runs a hybrid worker in Azure Automation runs the PowerShell on the local server
- 30:28to use sharegate to move the data from one site to another site
- 30:33because flow and logic apps couldn't natively handle
- 30:38it,
- 30:38nor it could even just running PowerShell in Azure automation.
- 30:43I think it's really the SharePoint endpoints,
- 30:46it's those APIs you start handling hitting throttling limits.
- 30:49That same article that talks about,
- 30:51and I completely lost it on all my browser windows when it talks about the
- 30:55limitations of moving, let's see if I can go pull it up here. Uh,
- 31:00moving and copying files across sites. So copy move operations.
- 31:04A single operation has three requirements.
- 31:06No more than a hundred gigabytes of total file size.
- 31:09No more than 30,000 files in each file must be less than 15 gigs.
- 31:13Now maybe maybe moving and copying down locally or using Azure
- 31:18Data Factory or something like that could work
- 31:23around it. But you're,
- 31:25I think you're still gonna start hitting thresholds and constraints with some of
- 31:29that stuff. That's probably your best bet.
- 31:32The other thought that is kind of in the back of my mind,
- 31:36but again it's still using rest calls and API endpoint that can get throttled
- 31:40would be just to use PowerShell to download the files at a SharePoint.
- 31:44Just write a huge PowerShell script or write one that you can point at a site
- 31:49and just go aside at a time, run the PowerShell script,
- 31:52download 'em all locally and then use AZ copy to move 'em up. Yeah.
- 31:55I, so, so that was another thought that I had was you could be slightly
- 32:00transactional in nature about this.
- 32:03So AZ copy has things like command line, pr, uh,
- 32:06parameters where you can point it at a say like a folder
- 32:11of files that are sitting locally for you.
- 32:14But ultimately what it has to do is enumerate those and every upload
- 32:19operation is gonna be kind of a, a separate call and a separate API call.
- 32:24So if you're processing a objects to a certain degree in a linear
- 32:29fashion, you know, you're processing them, you know, in a,
- 32:33in that way versus a highly concurrent model where maybe you're gonna process
- 32:37like 10 million at once kind of thing.
- 32:39Like that might be hard for you to pull off something like Casey copy.
- 32:43It would be a long running operation.
- 32:45But depending on where the files are hosted in SharePoint,
- 32:50like let's say it is a hundred sites,
- 32:53go into something like Azure Automation and create a runbook in Azure
- 32:58automation where you don't even have to use AZ copy.
- 33:01You could actually use something like just the Azure CLI or Azure PowerShell to
- 33:05do this as well. Like if you want to be PowerShell driven, like you said, hey,
- 33:09let's go ahead and download through PowerShell using the SharePoint APIs,
- 33:13whether you're doing that with PMP or just native kind of SPO commandlets and
- 33:18then just call an upload operation on the other side and script it out that way,
- 33:22right? Like call this site, get everything in it, download one, upload one,
- 33:27download one, upload one, and then concurrency.
- 33:29You'd have to figure out how to kind of spread that out.
- 33:32Like how do you get peanut butter all over the place so that you can get the
- 33:36scale and throughput that you want. Like it is kind of an interesting problem.
- 33:40I, I kind of thought about it as easy in my head again, I started with,
- 33:43it's got tb so it, so it's easy in it, right? Like earlier today the,
- 33:47the very first customer conversation that I was working with a customer on was a
- 33:52uh, six petabyte data lake . So I saw 200 terabytes. I was like,
- 33:57you're my easiest problem today, right? Like yeah, you got nothing going on.
- 34:01. Yeah. See and so you mentioned Azure automation.
- 34:04We may have done a podcast on this a long time ago.
- 34:07You would something this big,
- 34:09you would have to run it in VMs if you wanted to script it with Azure
- 34:12automation,
- 34:13it would have to be hybrid worker or you'd have to run it just natively a VM and
- 34:17Azure because Azure automation,
- 34:19you would start hitting the fair use limits where if you start consuming too
- 34:24much memory, too much CPU or your script runs for too long,
- 34:27they essentially boot you out of the queue and restart
- 34:32your job length of time.
- 34:35I've had it happen to me as quick as 30 minutes I've gotten away running
- 34:39something as long as two hours. This amount of files,
- 34:42you'd be way beyond two hours. And because you're dealing with files,
- 34:47I would assume your CPU and memory consumption in whatever's doing the
- 34:51processing and the backend, they boot you out way sooner than two hours.
- 34:55So I don't think you could even do this with Azure Automation successfully, uh,
- 34:59without doing a hybrid worker, without running it natively on a server. So,
- 35:02gotcha.
- 35:03So at that point you just orchestrated on a server anyway,
- 35:05it's just a power sell script, right? Run it, do the Rone thing,
- 35:09write it and then set it,
- 35:10forget it and let it run and do its thing in the background.
- 35:12I think that's what I would do. I think my approach would be scripted out,
- 35:17run it on a server. If you wanna run it in parallel to your point,
- 35:21if you have a hundred sites and you are okay spinning up a hundred servers in
- 35:25Azure, just spin up a bunch of servers, dedicate a server to it,
- 35:29it might help with some of the throttling.
- 35:32I don't know what they watch from the SharePoint side in terms of throttling
- 35:36when you're hitting the api, if you're coming from a hundred different servers,
- 35:40maybe you don't get throttled as quick cause you're coming from a bunch of
- 35:43different IP addresses.
- 35:44Maybe they recognize that it's coming from the same tenant and going to the same
- 35:49blob storage and they're gonna start throttling you.
- 35:51But I think PowerShell on a server to download it then AZ copy to upload it is
- 35:55probably your best bet. All right.
- 35:57Well I, I mean if anybody else has any good ideas, uh, , let,
- 36:01let us know.
- 36:01This is almost where I want like the bulk export tool from SharePoint. You know,
- 36:05like kind of like we have like Azure import export where I can just send them a
- 36:08disc or have them send me a disc.
- 36:10I almost want you to just be able to send me a disc of my stuff,
- 36:13just do a dump and, and send it out the other side, put it in the mail.
- 36:16And if it gets to me in two weeks, do.
- 36:18They have that?
- 36:19I don't know. I don't, I don't think,
- 36:20I don't think for SharePoint such a thing exists.
- 36:23There's bulk import in the form of these tools, right? Like,
- 36:27like SPMT and mover.io and some of the other things you mentioned,
- 36:31but I don't think there's a bulk export tool. Like,
- 36:36not even on the compliance side, right?
- 36:38There's nothing that I can say to go in and gimme all my files.
- 36:41I do believe I was, I was doing some quick Googling on the side,
- 36:43like I did find,
- 36:45like there's CIS tools makes a document downloader.
- 36:49So maybe there's somebody else out there that makes, you know,
- 36:53something similar if that wasn't your jam.
- 36:56But it does look like there are a couple,
- 36:57at least a few kind of paid tools out there that would let you do it.
- 37:02So yeah, I, I don't know. I thought it was an interesting question though. Yeah.
- 37:06Absolutely. As,
- 37:07and I would be curious if anybody else has any other insight into it.
- 37:10Because one, I've, to your point, I've never gotten this question before about,
- 37:14I have all this data, but I guess it kind of makes sense,
- 37:18like if you're looking to archive stuff,
- 37:20we had some episodes too where you talked about SharePoint as a files server.
- 37:23It's an expensive place to store a bunch of files if you're just storing
- 37:28them there and never touching them again.
- 37:31Like I can see where this could become a request.
- 37:35As companies create lots of data and they have older files,
- 37:38they don't delete 'em, they wanna archive 'em somewhere.
- 37:41It is way cheaper to store 200, a hundred, 300 whatever,
- 37:46hundreds of terabytes and blob storage than it is SharePoint.
- 37:50SharePoint you're paying, again, I figured it out. I did the math.
- 37:53You get 10 gigabytes for every user.
- 37:55So every user license gives you 10 gigabytes.
- 37:57One terabyte is a hundred user licenses.
- 38:00So you're either paying for a bunch of licenses if you're going out and buying
- 38:03file storage,
- 38:05it is still 20 cents per month per
- 38:10gigabyte.
- 38:1020 Cents per month per gigabyte. Eh, it starts to add up.
- 38:13It starts to add up compared to blob storage,
- 38:15especially if you can go into cool or archive.
- 38:17That's fra Is it like fractions of ascent? Yes.
- 38:20Yeah, .
- 38:21Yeah. SharePoint storage is 20 times more than archive blob storage.
- 38:26There could be a valid use case for this.
- 38:29Maybe Microsoft should go build a feature for archive to blob storage out of
- 38:33SharePoint when you get old files. That would be kind of a cool tool. .
- 38:38I I mean I, I certainly like the idea of it, right? Like, uh,
- 38:42pay me money, . That'd be, that'd be great. Yeah.
- 38:45Huh. Fascinating. All right. I'm curious, like Scott said, if you have feedback,
- 38:50let us know. Twitter, YouTube comments on the blog post,
- 38:54associating the podcast, whatever you want to do.
- 38:58If you're part of the membership,
- 38:59just give us feedback and Discord and we can share it on a future podcast. But.
- 39:04I was gonna say, if, if you were a a member,
- 39:06you could come along and participate as we record real time.
- 39:10You could see our yes,
- 39:11our made for Radio faces while we're here streaming off into the ether.
- 39:16Yes.
- 39:16Give us live feedback and input on all of our crazy ideas that we come up with.
- 39:21Indeed. All right. Oh, I think that's it for me today.
- 39:24I thought we'd get through that one in 15 minutes and here we are once
- 39:29again, you know, 35, 40 minutes into it and still debating.
- 39:34Still debating. We do have a question we need to get to it.
- 39:36We do have a question in Discord from one of our members that we have not
- 39:41gotten to that we should put it on the list, Scott,
- 39:45in the next week or two to answer that one cuz we can answer it in Discord.
- 39:49I've thought about going in and typing it up,
- 39:50but I think it's another interesting one that would be good discussion in a
- 39:54future episode.
- 39:55Next week. First thing,
- 39:57I'm gonna write it down on my Post-It note of things that I need to make Ben do,
- 40:02uh, we'll discuss. We would do it. Listener questions from Discord first.
- 40:07All right, sounds like a plan, Scott. And with that, go enjoy your weekend.
- 40:11Perfect.
- 40:11Thanks Beth.
- 40:14If you enjoyed the podcast, go leave us a five star rating in iTunes.
- 40:18It helps to get the word out so more IT pros can learn about Office 365 and
- 40:23Azure.
- 40:24If you have any questions you want us to address on the show or feedback about
- 40:28the show, feel free to reach out via our website, Twitter, or Facebook.
- 40:32Thanks again for listening and have a great day.