Latest / Microsoft Cloud IT Pro Podcast / Episode 360 – Who owns your data in Azure?
Transcript
- 0:03- Welcome to episode 360
- 0:05of the Microsoft Cloud IT Pro podcast
- 0:08recorded live on November 3rd, 2023.
- 0:12This is a show about Microsoft 365
- 0:14and Asher from the perspective of it pros
- 0:17and end users where we discuss a topic or recent news.
- 0:20And how about relates to you today,
- 0:23Ben Laments about a change in the data processing
- 0:26for Azure Sentinel, which side note
- 0:29less than two weeks later has now been reversed.
- 0:32And then Scott provides some suggestions
- 0:35around Microsoft's policies
- 0:37and how to keep up to date on them.
- 0:39Then they wrap up the show by talking about ways
- 0:41to track resource ownership over time
- 0:44for your Azure resources.
- 0:48Scott, I feel like we're maybe sort
- 0:49of back on schedule for a little bit.
- 0:51We've been kind of recording all over the place lately. Uh,
- 0:53- It is what it is.
- 0:54We'll get there. Holidays are coming
- 0:56- Up.
- 0:57Yeah, that'll throw us for a loop.
- 0:58We're only 15 minutes late today,
- 1:00- . Oh
- 1:02- Yeah, I got caught in a meeting about some stuff coming
- 1:05to the business, future announcements coming down the road,
- 1:09but I have an announcement.
- 1:11Let's just dive right into it.
- 1:12Scott, I have an announcement I got from Microsoft
- 1:14that actually was kind of disturbing
- 1:16and it was a little ironic
- 1:20with some other conversations I was having.
- 1:22So the other day I should find, I don't know
- 1:24where this conversation was, I was talking to a friend
- 1:28of mine and we were discussing,
- 1:31I think we were actually discussing browsers
- 1:34and our preference for browsers
- 1:36with Firefox versus Edge versus A and Safari
- 1:41and all the options. Brave all
- 1:43- Those, all the fun things. Yes.
- 1:45- And he made a comment about trying
- 1:47to start transitioning from Edge to Firefox.
- 1:50One, he's been having some issues playing videos and Edge,
- 1:53but the other comment he made was he's, I'm starting
- 1:55to get a little worried about this AI stuff with some
- 1:59of these companies training LLMs,
- 2:02and them using my data to train their models
- 2:05because there's also been some debate that we've had
- 2:08around this, about like training models off of website data.
- 2:13If some of these companies,
- 2:15not necessarily specifically Microsoft
- 2:17but the OpenAI models the bar, like all of these,
- 2:22how they're learning is are they
- 2:24potentially scraping like copyrighted material off
- 2:27of websites to train these Lang large language models?
- 2:31Like who owns the stuff on my blog if they start using my
- 2:35blog and answers for these large language models?
- 2:39All of that. So it was kind of a security discussion
- 2:42around AI and access to data and who you trust
- 2:45and who you don't trust and all of that.
- 2:47This was literally like three
- 2:49or four hours later I got an email titled Microsoft
- 2:53Sentinel will begin collecting security research data on
- 2:57forward December, 2023.
- 2:59So December 4th of this year,
- 3:01and this is not for my tenant, this is actually for one
- 3:04of my customer's tenants,
- 3:06but it says we're changing
- 3:07how we collect data Starting on this date.
- 3:09You are receiving this message
- 3:11because one of your Microsoft Sentinel workspaces,
- 3:13so log analytics essentially is opted out
- 3:18to share the data with Microsoft security research.
- 3:21This was turned on for a reason.
- 3:23We did not want this particular data being
- 3:26shared to Microsoft Research.
- 3:27. Turns out Microsoft security research protects our
- 3:30customers against threats and attacks
- 3:32by delivering built-in state-of-the-art detections,
- 3:34investigating tools, attack disruptions
- 3:36and mitigations,
- 3:39building more accurate models, AI models.
- 3:42I am reading between the lines there.
- 3:44It doesn't specifically mention AI requires diverse data
- 3:47sets and signals because of this.
- 3:50Starting December 4th, 2023,
- 3:52Microsoft security research will have access to your data
- 3:55to help build, test and optimize analytics
- 3:57models and detections.
- 3:59Okay, so I turned this on specifically
- 4:02because I wanted this data to stay private
- 4:04and now you're telling me
- 4:06- And by turned it on, you mean turned
- 4:07- It off - Explicitly opted out Out of this thing. Yeah,
- 4:11- Out of this.
- 4:12And now they're just saying, sorry, too bad. We don't care.
- 4:16We need to build our models
- 4:17so we're gonna go grab your data.
- 4:18Anyways, they do go on to say,
- 4:20so then they have some more information
- 4:23and I haven't dug into this too far yet, but it's privacy
- 4:27and data handling commitments
- 4:28and the Microsoft online services terms
- 4:30and online services data protection addendum,
- 4:33their legally applies.
- 4:35It also says customers using customer managed keys CMK won't
- 4:39be affected by this change.
- 4:40So I am theorizing that by
- 4:43that single sentence down towards the bottom
- 4:45that if I go switch everything to customer managed keys,
- 4:48it's not gonna get shared
- 4:50because potentially this is encrypted then
- 4:53beyond their ability to grab it.
- 4:54And then it also has a help
- 4:55and support if this change affects your environments.
- 4:58Well yeah, 'cause you're actually changing settings on me
- 5:02and you'd like to learn more about it.
- 5:04Please contact support
- 5:05and then it just walks through like the path to go
- 5:08through under your issue type and support NA Azure.
- 5:11Yep. And then it gives the account information,
- 5:14subscription name, all of that in the
- 5:15Azure subscription that this is using.
- 5:17I'm not a big fan of this.
- 5:18I get if they wanna do this and they send an email
- 5:21and ask permission or say would you like to help?
- 5:26Or even saying,
- 5:29I mean it would probably be borderline is if you don't do
- 5:32anything, this will be turned on.
- 5:33But you can go back in
- 5:34and toggle this checkbox to leave it off
- 5:36or something along those lines.
- 5:39But just kind of blatantly saying, we don't care
- 5:42what you said before about sharing this data,
- 5:44we're gonna go get it anyways
- 5:45because we want to train our large language models better.
- 5:48I totally get the how it helps with security,
- 5:50how it helps building the models
- 5:52with security co-pilot coming out.
- 5:54I'm assuming that also has something to do with this this,
- 5:57but just going in and blatantly saying
- 6:00now we're gonna access it.
- 6:01Especially when they do tend
- 6:04to focus a little bit more on security than maybe some
- 6:06of the other big players.
- 6:08I've tended to trust Microsoft more than Google when it
- 6:11comes to keeping my data secure and the privacy of my data.
- 6:15But this just kind of,
- 6:17it rubbed me the wrong way when I saw this
- 6:19- .
- 6:22Yeah, so I'm, I'm over here shaking my head and
- 6:25and just rubbing my eyes.
- 6:26So there's a couple of things.
- 6:30So um, I think part of it is the
- 6:34just abject fear that's been instilled in everybody.
- 6:37Like I would not want an LLM hallucinating on top
- 6:41of my data either back to me
- 6:43or certainly in context of potentially like my data
- 6:46with other customer usage, things like that.
- 6:49So like reading between the lines
- 6:52and I'm not a lawyer,
- 6:54I don't work on the sentinel team, anything like that.
- 6:56A lot of sentinel isn't necessarily AI driven
- 6:59but it is ML driven which ultimately like these large
- 7:02language models Yeah those are actually ML as well.
- 7:05Like we've conflated the whole
- 7:07ML AI thing, blah blah blah, all that stuff.
- 7:08Yep. I think it's good to have the signals in there.
- 7:11I think it's bad to kind of revert on a decision like that.
- 7:14It's a good reminder to everybody that when you sign up
- 7:19for these services, like there is a just absolute
- 7:24crap ton of legalese under underneath the covers
- 7:27that drives these things
- 7:29and I don't think enough customers
- 7:32actually go and read those.
- 7:35I've had the opportunity a couple times in over the course
- 7:39of my time at Microsoft to actually have
- 7:41to go read up on the legal agreement for
- 7:45Microsoft customers when it comes to the what's known
- 7:49as the mosa or the Microsoft Online subscription agreement.
- 7:54So that governs things for MCA customers, sorry,
- 7:58Microsoft customer agreement MCA
- 8:00and you know EA customers enterprise agreement customers
- 8:04have a slightly different set
- 8:05of terms even if you don't use Azure,
- 8:07there's always just the general like Microsoft terms of use
- 8:10that you're using in tools, things like that along the way.
- 8:14So there's really two things
- 8:16that even if you're not into the legalese
- 8:18of the stuff you're doing, it's kind
- 8:20of interesting if you can go
- 8:21and stomach your way through it.
- 8:22One is that most Microsoft online subscription agreement
- 8:26and then the thing that everybody always forgets is the DPA,
- 8:30which is the data protection addendum.
- 8:34So the DPA in many cases is actually the thing
- 8:40that drives the contract.
- 8:43Like the legal contract between you
- 8:44and Microsoft particularly with regard to
- 8:48the processing and the security of your customer data
- 8:52and your personal data in there.
- 8:55In your case here. Yeah, you've got a couple options.
- 8:58One is go familiarize yourself with the DPA.
- 9:02And again I encourage every customer to actually do that.
- 9:05Like not just your lawyers but folks should go
- 9:07and read it, go read the DPA
- 9:08and see what Microsoft does with your data.
- 9:10I don't think Microsoft is like, they're definitely not
- 9:13as clear as companies like
- 9:15Apple when it comes to this stuff.
- 9:16Like Apple's very clear, hey we do processing on device,
- 9:20here's why we do it on device, blah blah blah privacy, all
- 9:23that kind of stuff with providers like Google, Microsoft,
- 9:27Amazon, like all that stuff happens in the cloud
- 9:31or the the other option like you said is yeah flip over
- 9:34to CMK.
- 9:36So CMK is customer managed key
- 9:38where effectively you're managing your encryption key
- 9:42including things like rotation of that encryption key
- 9:46inside something like Azure key vault so that you are in
- 9:50full control end to end and you have that capability.
- 9:54So I run into a lot of customers, I don't,
- 9:56like I said I don't work on cental
- 9:58but particularly in like storage land,
- 9:59like people are very sensitive to this kind of thing.
- 10:02So we do like customer provided key.
- 10:04So we actually we we do customer managed key
- 10:07where customers can manage keys inside of KeyVault
- 10:10but we also have this capability called customer
- 10:13provided key where per request,
- 10:15like per upload you can actually send a different encryption
- 10:19key for every single object if you're like really
- 10:23super kind of paranoid about that thing
- 10:27and how it comes together and what that looks like.
- 10:30The other thing that the DPA is helpful for
- 10:35is it also doesn't just govern how Microsoft uses your data.
- 10:39It governs how things like data destruction go
- 10:43and you know
- 10:44what happens when you know you issue a delete request
- 10:48or you delete a particular resource
- 10:50or something like that along the way.
- 10:53And then for this one you're actually kind of lucky in
- 10:57that you are in maybe like an Azure service here.
- 11:01Like your encryption capabilities
- 11:02between services are going to vary.
- 11:05It's a little bit different over in Office 365 M 365 land.
- 11:09It's different over in Dynamics land.
- 11:11You might not necessarily be able to come to
- 11:15consensus across all of those things,
- 11:17especially when it comes to some
- 11:19of the large language models you were talking about.
- 11:21Like those rolling out and getting to where they need to be.
- 11:24Like they're just gonna be baked into experiences in some
- 11:27places and customers might not have the level of control
- 11:32that they necessarily want there upfront.
- 11:34And that becomes part of the rationalization exercise
- 11:37of just trying to figure out like is this
- 11:39the thing that I want to do?
- 11:40Is it the right thing
- 11:42for me at this given place time? That kind of thing.
- 11:46- Definitely and maybe it has this
- 11:48and I have not probably spent as much time reading these
- 11:53DPA and legalese as I should,
- 11:57but the other thing that would've been nice to include
- 12:01in either the email
- 12:02or have a link to it is what data is actually being sent?
- 12:07Like is it row for row
- 12:10data from my tables in Sentinel
- 12:13or is it things like when Sentinel detects a brute force
- 12:17attack, the IP addresses of a potential brute force attack
- 12:21and where is it almost anonymized I guess in sense
- 12:25or sent a summary of the data that's coming in versus
- 12:30this is sending like IP addresses and usernames
- 12:35and countries
- 12:37and I don't know 100% at least from this email,
- 12:43what does it mean when this data's being
- 12:46provided now to the Microsoft security research in terms
- 12:50of the level of data or the details around what that is.
- 12:53- That's an interesting one.
- 12:55I actually, I pulled up the DPA while you were chatting just
- 12:59to see is there anything specific to
- 13:02to Sentinel in there two that's called out
- 13:04and yeah there's not anything in that one.
- 13:08The other thing it's gotten kind of harder over time
- 13:11to find some misinformation like the DPA a used
- 13:14to be like an online thing
- 13:15or one of 'em did the online service terms
- 13:18and even like SLAs
- 13:20and things now they're like turning into like PDFs
- 13:21and word docs so you have to like explicitly go out
- 13:24and download them and figure all that stuff
- 13:28- Out.
- 13:29Yeah it's interesting.
- 13:30And then this is a conversation kind of going along side
- 13:34of this in Discord too is you mentioned this is Azure,
- 13:37this also comes into play as
- 13:39what this would mean potentially
- 13:41for Microsoft 365 down the road.
- 13:44This particular case isn't applicable to Microsoft 365
- 13:48and Azure the customer managed key
- 13:52isn't necessarily a expensive paid
- 13:56for service versus Microsoft 365 I believe if you wanna do
- 13:59like a customer managed key and Microsoft 365
- 14:03but you need to be on an E five license
- 14:06and I can't remember which E five it is,
- 14:08if it's the EMSE five
- 14:10or the Office 365 E five,
- 14:13Microsoft 365 E five would definitely cover it
- 14:15'cause it has both of those in there.
- 14:16But if I remember right,
- 14:18customer managed keys in Microsoft 365 or E five.
- 14:21So if you ended up in this boat
- 14:23with say Microsoft 365 copilot as that's coming out
- 14:26where you got an email saying you've elected
- 14:29to not share certain information with the Microsoft 365
- 14:34security research and the only way
- 14:36to block it is customer managed keys.
- 14:38You could find yourself in a position where the only way
- 14:41to stop that would be to upgrade from like
- 14:44that E three five plan 2 85 plan.
- 14:47The whole large language model.
- 14:50Again, to your point, it it's my data
- 14:52but it isn't Microsoft's data center.
- 14:54They're hosting it all.
- 14:55And again I haven't read all of the legalese
- 14:58around when they can or can't do this type of stuff
- 15:00but it definitely does lead to some interesting things
- 15:05to consider as everybody is now scrambling for data
- 15:10for these large language models where
- 15:12before part of my thing was Google always wanted your data
- 15:15and I'm gonna call out Google specifically
- 15:17because they're in the advertising business.
- 15:19Facebook or Meta always wanted your data
- 15:21'cause they're in the advertising business.
- 15:23They benefit from getting as much data as they can about you
- 15:27to better advertise and market to you.
- 15:29Where let's face it, that wasn't Microsoft's business
- 15:32but now with AI
- 15:33and everybody kind of battling
- 15:35for these large language models
- 15:37and the training of 'em, all of a sudden
- 15:40I find Microsoft wanting a lot more data than maybe they did
- 15:44before so that they can train these models.
- 15:46Which definitely leads to, like someone said,
- 15:50it's some interesting days, interesting times
- 15:52to think about some of the implications of this
- 15:54beyond just copilot can help me with my work. Now
- 15:57- I hate to be a stickler for language
- 15:59but I, I kind of would continue to encourage you to
- 16:03not think of everything in ML land as
- 16:06a large language model.
- 16:08So like large language models are specific things like in
- 16:11many cases like in the case of chat GPT we're dealing
- 16:15with language transformers.
- 16:16So these are things, these are machine learning models
- 16:19that are built around natural language processing.
- 16:23So if you're ever played around with say like Lewis
- 16:26and sorry the language understanding intelligence service
- 16:30and building out like chatbots in Azure
- 16:33or something like that, you're dealing
- 16:34with natural language processing.
- 16:36There's all sorts of other ML models that can
- 16:41and do exist out there.
- 16:44So that could be things like you said like maybe maybe I'm a
- 16:49somebody like Google and I'm trying to figure out spread of
- 16:54advertising and cohorts of customers
- 16:56and so I can give my advertisers the best experience
- 16:59where they can target their ads
- 17:00to people in a certain demographic.
- 17:01Like I only want to target 25 to 30 year olds that live
- 17:07in Spain or even a certain region.
- 17:10Like you can get down to that level
- 17:12of granularity sometimes
- 17:13depending on what's going on out there.
- 17:14And that's very different than hey I'm gonna throw a large
- 17:18language model against this thing
- 17:19or some type of transformer against it so it can ingest
- 17:22and do a bunch of natural language processing.
- 17:24So I would bet in the case of something like Sentinel,
- 17:28'cause Sentinel is a product
- 17:29that like you can bring your own ML models to as well.
- 17:32Like you can do your own inference on top of it even on top
- 17:36of like your own custom data
- 17:37that you potentially bring into Sentinel,
- 17:40say you're doing like taxi or things like that
- 17:42and kind of side curing your own data in there.
- 17:45So I would bet
- 17:46that it's more about all up signals especially
- 17:49'cause if you look at the way things have been going
- 17:51with security in general like across the tech companies,
- 17:56I don't think Microsoft's any exception here.
- 17:58The more signals you have the better off you can be. Yep.
- 18:01What's the sensitivity like just a devil's advocate,
- 18:04like what's the sensitivity to a signal in Sentinel versus
- 18:09something in your sign-in logs in Azure AD like being
- 18:14anonymized and correlated across the service, right?
- 18:17Microsoft always talks a lot about
- 18:19hey why does security work so well in
- 18:22ID the artist formerly known as Azure ad?
- 18:25It's because they have billions of logins
- 18:27and billions of signals that they can run that inference on
- 18:31and figure out the next thing there.
- 18:37- Do you feel overwhelmed by trying
- 18:38to manage your Office 365 environment?
- 18:41Are you facing unexpected issues
- 18:42that disrupt your company's productivity?
- 18:44Intelligent is here to help much like you take your car
- 18:47to the mechanic that has specialized knowledge on how
- 18:50to best keep your car running Intelligent helps you
- 18:53with your Microsoft cloud environment
- 18:54because that's their expertise.
- 18:56Intelligent keeps up
- 18:58with the latest updates in the Microsoft cloud
- 18:59to help keep your business running smoothly
- 19:01and ahead of the curve.
- 19:03Whether you are a small organization
- 19:05with just a few users up to an organization
- 19:07of several thousand employees they want to partner with you
- 19:10to implement and administer your Microsoft Cloud technology,
- 19:14visit them at intelligent.com/podcast,
- 19:18that's I-N-T-E-L-I-G-I-N
- 19:23k.com/podcast for more information
- 19:26or to schedule a 30 minute call
- 19:28to get started with them today.
- 19:30Remember intelligent focuses on the Microsoft cloud
- 19:33so you can focus on your business.
- 19:37I think that's the part that I get where to your point,
- 19:40Sentinel would absolutely, I was trying to find a tweet
- 19:44but I think I lost it in our chats, being able
- 19:46to just pull those signals
- 19:47and if they're anonymized enough, I totally get it.
- 19:50Like you look at the amount of signals that Microsoft gets
- 19:53and I've talked to red team companies
- 19:57before that are actually writing viruses trying
- 20:01to bypass things like
- 20:03Microsoft Defender from an antivirus perspective
- 20:06and they've told me,
- 20:08and I don't have any validation
- 20:10for this other than just word of mouth talking
- 20:12to these people that Microsoft Defender antivirus is one
- 20:15of the hardest antivirus software to bypass
- 20:20when they're just writing a virus from scratch
- 20:22and that once they are able to write one
- 20:25Microsoft Defender is also one of the ones
- 20:27that shuts it down the quickest.
- 20:29They said I essentially get one shot at it, I deploy
- 20:31that virus, I use it, I get in, I never works again.
- 20:35And I think some of that is to your point of
- 20:38Microsoft is able to take all
- 20:40of these different signals from their products
- 20:44and they use them to protect other tenants.
- 20:48So myself, anybody else listening to this podcast
- 20:52that uses Microsoft 365 is benefiting from the signals
- 20:55that these Microsoft security products are getting from
- 20:58people also trying to attack Microsoft.
- 21:01You look at what Microsoft 365 is, 90%
- 21:04of the Fortune five hundreds, how many
- 21:06of them are deploying Defender Sentinel?
- 21:09Some of those. But there's signals coming in from those
- 21:11that Microsoft uses to protect the small guys as well.
- 21:15So I also do understand that if those are anonymized enough,
- 21:19if they can figure out those signals
- 21:21and this is where I, maybe you have
- 21:24to trust Microsoft a little bit that they're not
- 21:27grabbing a bunch of PII type of data
- 21:29or really sensitive internal data
- 21:33but looking at some of those signals
- 21:34to really help strengthen that.
- 21:36I think there absolutely is a benefit to that.
- 21:40I think it was a tweet from Merrill
- 21:41that I saw about the amount of signals
- 21:43that Microsoft does get and analyze a day
- 21:45and it's like in the trillions I believe
- 21:49of signals a day that Microsoft gets around
- 21:54people trying to breach the Microsoft products essentially.
- 21:59Or maybe it was even specific to Entra. It's
- 22:01- A ton of data, right?
- 22:02If you consider like particularly in the case of intra
- 22:04and login data 'cause Microsoft is able to aggregate
- 22:09and correlate signals across the public service as well.
- 22:13So like your endpoint when you log to enterra
- 22:17login do microsoft online.com, like that's the same thing
- 22:21that MSAs and all that use and all that use as well.
- 22:25I don't know if you have any options here so
- 22:28I'll put a link in the show notes.
- 22:29So one thing that does come up from time
- 22:31to time is handling personal data in things
- 22:34like log analytics.
- 22:35Like I haven't seen specific guidance for Sentinel
- 22:38and I've been kind of like binging Duck Tuck going
- 22:40and then googling in the background here.
- 22:42uh, I couldn't find much
- 22:43but there is specific guidance out there for log analytics
- 22:46and app insights customers.
- 22:48So maybe you want to take a look through that as well.
- 22:50I don't know if there's things that can potentially like
- 22:52make it a little bit better by potentially getting
- 22:55that data over to a different log analytics workspace once
- 22:58you're done with analysis in sentinels
- 23:00so you can still retain it for yourself.
- 23:02Are there potential retention things you wanna do At the end
- 23:05of the day the data's the data though, right?
- 23:08So you've gotta kind
- 23:09of figure out your path to navigate through there.
- 23:11Whether that's saying well sentinels not the service
- 23:14for me given a set of constraints that exist
- 23:19or it it is and I'm gonna live with it
- 23:21or sentinel's not the thing for me in its current form.
- 23:24So I need to do something
- 23:26and potentially upgrade to some form
- 23:28of my own encryption key like customer managed key
- 23:30and then live with any licensing implications that come
- 23:34with it to have that associated level of privacy. Yeah
- 23:38- I found the tweet, it was not about signals,
- 23:41this was just about password attacks per second
- 23:44blocked 4,000 password attacks per second this year.
- 23:48Like that's just an insane amount of data coming in
- 23:52because it's not just that they blocked 'em,
- 23:54I'm sure they were looking where they're coming
- 23:55from, all of that.
- 23:57But any who, I'll have to go look through that article
- 24:01as well and if we want to make any changes to log analytics
- 24:06and how that data's in there, the
- 24:08- Nice thing on your side is that you work
- 24:10with enough customers that it's also like the all learning
- 24:13that you can carry forward for them as well
- 24:16and say okay here's potential considerations.
- 24:20We might wanna think about this as we're turning it on.
- 24:23So you can kind of front load the conversation
- 24:25with new customers in this like crazy always
- 24:30ever changing landscape.
- 24:32- Exactly. So interesting scenario.
- 24:35Do you have other, you had
- 24:36something else you wanna talk about today?
- 24:38We spent a long time on that
- 24:39but let's dive into your topic. We
- 24:41- Did, so mine's a little bit of a a logging thing as well.
- 24:46So this is one that came up
- 24:47with in in a customer conversation with me recently
- 24:51and I think it impacts a lot
- 24:54of customers in Azure or
- 24:57or there's the potential for customers to ask this question.
- 25:00I don't know, it's like a big impacting issue
- 25:01but so have you ever been in the boat where you're working
- 25:05with somebody and you sit there
- 25:08and you do a bunch of deployments
- 25:10and you kind of turn stuff on
- 25:11and then you walk away from it for a while
- 25:13and then you move on to the next thing
- 25:16and then the next person comes in behind you
- 25:18and they say, huh, what is this thing?
- 25:20Who created it? Do I still need it anymore?
- 25:23Does it need to be on, can I delete it?
- 25:26And it's one of those things that you came up I,
- 25:29I was talking with a customer funny enough about some
- 25:31storage accounts and they're like, Hey I have all these,
- 25:33they had a couple hundred different accounts out there
- 25:35and they're like, can I delete them?
- 25:37Like well do they have activity ?
- 25:40Have you checked just the metrics
- 25:41to see if you're driving any transactions in there?
- 25:45Oh no I haven't done that. All right, well let's go do that.
- 25:49Let's spin up an easy mod workbook
- 25:50and uh, Azure monitor workbook
- 25:52and we'll look at some of this stuff and
- 25:53and see what's going on.
- 25:55Oh I see these 10 accounts over here
- 25:57don't have any activity.
- 25:58They go can I delete 'em?
- 26:00I don't know what, what data's in there. Well I don't know.
- 26:03Maybe I should go and talk to the person who manages it.
- 26:06So the crux of the question is
- 26:08who actually manages your re Yeah,
- 26:10who manages resources in Azure
- 26:13and how do you go back in time
- 26:15and figure that out for resources that
- 26:18have potentially existed for years and years and years?
- 26:23- A hundred percent. And it's funny you bring this up
- 26:26'cause I had a customer doing this exercise as well
- 26:28that I'm supposed to actually go do some work trying to sort
- 26:32through resources and figure some of this out
- 26:36because they're in the boat
- 26:37where they have everything in five subscriptions
- 26:39and we need to break it out into 14 subscriptions
- 26:42but we don't wanna move stuff that's not being used anymore.
- 26:45I mean this is not an insignificant
- 26:46amount of resources either.
- 26:48We're talking like thousands of resources.
- 26:50It's not quite, I don't think it's
- 26:52up in the tens of thousands.
- 26:53It might be definitely not hundreds of thousands
- 26:55but to your point it's okay we have these five
- 26:58subscriptions, thousands and thousands of resources first
- 27:01before even who owns them
- 27:03or who would know is how do we even go through
- 27:07and figure out which of these resources are still used,
- 27:10which can be deleted, which maybe can we archive?
- 27:13Are there resources that we could change the SKU on,
- 27:17have SKUs changed so that we can get some price savings
- 27:20by making some changes to 'em?
- 27:22And to your point then, who actually is in charge
- 27:25of making those decisions?
- 27:27Because some of these are production services, some
- 27:30of those are dev, some of those are test
- 27:32and what's going on.
- 27:34So since I have that question, what's the answer Scott? I
- 27:37- Had a couple of thoughts here
- 27:39and some of it is we're eventually we're gonna hit a
- 27:42roadblock and we can't go back in time all the way.
- 27:44So first thing is, is this resource being
- 27:48actively used right now?
- 27:50I think one of the best ways to do that is
- 27:53start over in Azure Monitor.
- 27:56So I often refer to that in shorthand as azon
- 28:01and over in Azon there's a bunch of built-in workbooks
- 28:05around kind of core service metrics.
- 28:08So if you're dealing with a core service,
- 28:10so I'm thinking things that fall into kind of compute,
- 28:14networking, storage, many of the data analytics services,
- 28:17things like that, they all have a set
- 28:20of default metrics which often drive back to things
- 28:25like transactions in the service, right?
- 28:27If I'm dealing with a storage account, am I seeing puts
- 28:30and gets actively to that storage account?
- 28:32And if I am, I I I know somebody out there is using it
- 28:36and potentially that helps me take the next step
- 28:39to go and answer the question.
- 28:41Okay, now I know it's on who is using it
- 28:44and who is using it can go a couple different ways.
- 28:47If I'm dealing say with if virtual machine
- 28:51and who is using it, I might want to go
- 28:55and just check out the activity log for that VM
- 28:58or spit out some diagnostics
- 29:00and do something like log analytics
- 29:01or a storage account, some service like that and take a look
- 29:05and see okay great, I know this thing's powered up
- 29:07and it's on am I seeing active logins to it?
- 29:10Can I actually infer anything from a login name
- 29:14or like we talked about baston last week,
- 29:16a bastion connection, things like that.
- 29:18If it's a storage account I have resource
- 29:21logs that I can turn on.
- 29:22Most customers tend to complain once we get down
- 29:25to this step
- 29:26where we're saying like well go look at like the fine
- 29:28grained resource logs for the thing.
- 29:30'cause they go Oh well I gotta turn that on,
- 29:32I gotta configure it and it costs me money
- 29:34for the logs .
- 29:35Yeah I get it. Time is money
- 29:36and turns out when you use additional
- 29:39resources, those cost money as well.
- 29:40We're not, not talking about keeping
- 29:41this stuff on forever, right?
- 29:43You could even turn on diagnostic logs on a service for
- 29:4610 minutes, whatever a period of time is
- 29:48and just kind of see what you get in there.
- 29:50It doesn't need to be like tens of thousands of dollars
- 29:54or hundreds or even tens of dollars to to spend
- 29:57that stuff up and turn it on
- 29:59if none of that gets you anywhere.
- 30:01The next place that I would kind of go off to is
- 30:06the Azure activity log.
- 30:09So by default at the subscription level there is a set
- 30:14of information that's retained when management
- 30:17plane operations happen.
- 30:19So basically when you like fire off a request against the
- 30:22arm APIs, which the portal is doing all the time,
- 30:25like a create event for a resource, an update event
- 30:28for a resource, things like that.
- 30:29Those will all be in the activity log there.
- 30:33There's some limitations there.
- 30:34Like the activity log only logs three months
- 30:37of data by default.
- 30:38Unless you've gone ahead
- 30:40and configured like diagnostic export for that
- 30:43and potentially sent it out to another place.
- 30:45But I look at that as a little bit of a learning exercise.
- 30:48Hey here's another opportunity for you to go
- 30:50through if this is something that's important for you
- 30:52to audit in your environment, you really might want
- 30:54to think about turning on activity log exporting
- 30:57for the actual Azure monitor activity log
- 31:00and getting that data out
- 31:02to something like a log analytics workspace so
- 31:05that you can have a richer interaction
- 31:08to go back in time over that resource. I would
- 31:11- Agree.
- 31:12And that's kind of the approach we've taken
- 31:13as like you said it's log analytics,
- 31:16it's Azure activity logs, it's really just going
- 31:19through all the logging and looking at what's there.
- 31:22This is gonna help with the activity.
- 31:24I don't know that it always helps with the owner
- 31:27and I think this is a,
- 31:28- So the nice thing is with activity often comes identity
- 31:32because users have to authenticate
- 31:34to these things like hey we have
- 31:36to figure out if you're authorized and
- 31:37before we figure out if you're
- 31:38authorized we need to know who you are.
- 31:40Yep. To do it. So lots of that stuff is just happening in
- 31:43inside of OAuth
- 31:44and it's all authenticated against Azure AD anyway.
- 31:47So AD potentially becomes a source of logs for you.
- 31:51Okay, am I seeing sign in logs like
- 31:53during a a given period given time
- 31:55but as long as you have the log like it'll have a goid
- 31:59or something else in it that ties you back to
- 32:02a user identity over in Azure active directory which
- 32:07ultimately like in inferences and
- 32:09and gets you down to ownership Now all this stuff only helps
- 32:13you like if you have the data there.
- 32:15The other thing that I had mentioned
- 32:18and it came up in the chat over here on Discord
- 32:21was there's other things
- 32:22that you should just think about maybe from like a
- 32:23governance perspective, like there's some best practices
- 32:26around things like tagging your resources,
- 32:30be it either tagging a resource group
- 32:32or tagging individual resources in resource groups in Azure.
- 32:36Like you might wanna tag those with a,
- 32:38maybe not an individual user
- 32:40but like a responsible group for it.
- 32:43Or quite often like I encourage a lot
- 32:45of customers in like multi-home departments,
- 32:47like they run multiple departments,
- 32:48they wanna do chargebacks and stuff like that.
- 32:50Hey let's rationalize not just the
- 32:52environment this thing sits in.
- 32:53Is it prod, is it dev test, is it staging?
- 32:56But let's actually tie this back
- 32:58to something like an IO code
- 33:00or some type of like internal finance code that you have
- 33:04or even like just a department, does this belong to finance?
- 33:07Does it belong to you Azure?
- 33:09Does it belong to finance, hr, anything like that. Yeah,
- 33:12- I was gonna bring that up as well.
- 33:15Going through this exercise
- 33:16and what we talked about logs helps you maybe find
- 33:19what happened before.
- 33:20This is also a good opportunity I think to go through some
- 33:23of what you just said is maybe this is a good time
- 33:26to start thinking about tagging
- 33:28and even using Azure policies to enforce that
- 33:32a certain resource is tagged with a owner.
- 33:35And again whether that be a person, a department tagged with
- 33:39what type of resource this is this something
- 33:41that's getting stood up for test or dev
- 33:43or is this a production level resource?
- 33:46I've seen customers tag dates of when this was created,
- 33:50but thinking through maybe some of that additional metadata
- 33:54that you should start adding
- 33:55or do wanna be adding onto your Azure resources so that
- 33:59as you encounter this scenario again
- 34:01or as you move down the road
- 34:03and you wanna six months from now,
- 34:05a year from now do a regular type
- 34:08of audit of what's out there.
- 34:09Do we still need it? Is it still being used?
- 34:11Having some of those tags can also be very beneficial
- 34:16but you may not have them the first time you do.
- 34:18It might be something
- 34:19that says maybe we should start doing this.
- 34:21- I think it's one of those things like when you run into it
- 34:23you're like oh maybe I should start
- 34:24doing this now kind of thing.
- 34:26Yep. I see a lot of customers when it comes to tags
- 34:28that they tend to just get stuck in the rutt
- 34:33of trying to rationalize like all the options.
- 34:36'cause it's basically just name value pairs
- 34:39and you can be right,
- 34:41very freeform in your name values, things like that.
- 34:43There's actually some decent guidance out there in the cloud
- 34:46adoption framework about just like getting started with tags
- 34:49and like defining your strategy
- 34:51where it calls out a lot of these things.
- 34:53Okay, you might want to tag like the uh, department
- 34:56that owns the resource you might wanna tag like the
- 34:59operations team that owns it.
- 35:01I think that's very helpful like in the context
- 35:02of even if I don't have the individual owner,
- 35:05hopefully the people who operate the
- 35:07thing have a contact right?
- 35:08And they can get back to it
- 35:09and do it cost codes as well, like whether
- 35:12that's like an IO code
- 35:14or so like you know whatever it is that ties you back
- 35:16to a cost center within your
- 35:18organization is also super helpful.
- 35:20Not just for like chargeback
- 35:21but for hey where's the throat that I need to go choke
- 35:24to figure out the answer to this thing.
- 35:27I've also seen customers that do things
- 35:30and it is in the adoption framework
- 35:32where they tag things like the owner, the requester
- 35:35and they actually put email addresses in there.
- 35:37I'm kind of sensitive to that stuff sometimes.
- 35:39I don't know that you should 'cause then you're pumping like
- 35:42PII out potentially into things
- 35:44and then we're back to that whole
- 35:46what does Microsoft do with my data?
- 35:48Like did I pump those activity logs out to log analytics?
- 35:51Are there like Sam a company
- 35:53and I'm doing business in Europe.
- 35:54Are there GDPR implications for me or
- 35:57or things like that there.
- 35:58So your mileage may vary in like how far
- 36:01and how granular you wanna go,
- 36:03but I think it's definitely a good idea to sit down
- 36:05and just do that initial rationalization.
- 36:09And if you haven't done it today
- 36:10and you're just onboarding, hey now's a great time to do it.
- 36:13If you haven't done it
- 36:14and you're already onboarded,
- 36:15hey now's a great time to do it.
- 36:17- . Sounds good. I don't think
- 36:18I have anything else on that topic.
- 36:20Anything else you wanna cover on tagging,
- 36:21cleaning up Azure resources, deleting everything.
- 36:24I mean there's always the approach of you delete it all
- 36:26and see who screams, right?
- 36:28Make sure you backups on delete it
- 36:29and whoever screams is the owner there
- 36:31- Is that as well?
- 36:32Yeah, to just take it and
- 36:35and turn this thing over a different
- 36:37way and see what happens.
- 36:39And you might have some options there if you're dealing
- 36:41with a website, like you might not have to delete it,
- 36:44you can just shut it off or a database.
- 36:46That's certainly a way to do those kinds of things.
- 36:49Just make sure you leave it off long enough
- 36:51that you give people enough time to scream.
- 36:54Like I've seen people do things like shut off a database
- 36:58and they leave it off for a day or two
- 37:00and they're like, well nobody
- 37:01said anything so I'll delete it.
- 37:02But it turns out that database was only used on like month
- 37:04close or like closing out a quarter, even the end
- 37:07of the year and three months later they get, they get
- 37:10that pinging that says,
- 37:11oh hey, what happened to blah blah blah.
- 37:12And you're like, oh crap, I deleted that.
- 37:14Nobody ever talked to me.
- 37:15So yeah, you do kind of have
- 37:17to think about those things as well
- 37:18- For sure.
- 37:19Awesome. Well thanks Scott.
- 37:20It was interesting discussion today.
- 37:24- I got one more for you. Yes.
- 37:25If anybody stuck around this long, we are back in November.
- 37:30So last November we did this thing where we tried
- 37:32to raise some money for Girls Who code.
- 37:35- Oh yeah. - And we had uh, a bunch
- 37:38of awesome listeners donate and we raised $1,500 this year.
- 37:42I'd like to see if we can raise 2000 and get out there
- 37:45and help the next generation of IT pros
- 37:49or developers, whatever that happens to be
- 37:52with Girls who code.
- 37:53So I'm gonna be harping on this one
- 37:55for the next couple weeks just
- 37:57to see if we can get people
- 37:58going during this season of giving. Yes.
- 38:00- So we should do that. Should we
- 38:01extend this longer than November?
- 38:02Because we are also recording with some of our schedule
- 38:07for those that are still sticking around.
- 38:09You actually won't hear this until November
- 38:1316 if I'm doing my date math right. So
- 38:16- Here's the thing, it's open all year , so
- 38:18- Should, that's true. You
- 38:19- Don't have to wait and come in. But we
- 38:21- Need to set a sense of urgency.
- 38:23We need to set a time like we wanna get it by this time.
- 38:26Should we see if end of the year,
- 38:28should we go through the end of the year?
- 38:30- I forgot about the lag in release.
- 38:31Yeah, we'll go to the end of the year. That works for me.
- 38:34- See if we can get $2,000 by the end of 2023.
- 38:37By the time people first hear this, that gives you a month
- 38:41and a half through the Thanksgiving holiday season. Yep.
- 38:45- Should you celebrate, but yes,
- 38:47during November, December,
- 38:48whatever that happens to be. So yes,
- 38:49- That's true.
- 38:50Thanksgiving, I was talking to my kids about that.
- 38:51They were like, what holidays are worldwide versus just the
- 38:55us And there's really only a couple I can think of that are,
- 38:59well there's no, there's a handful,
- 39:02but there's a lot of them that are just localized
- 39:04to countries.
- 39:06Thanksgiving being one of those. Yes. Yeah.
- 39:09Anyways, we do not need to go on the holiday rabbit hole
- 39:11today, but through the end of 2023,
- 39:14$2,000 for Girls Who Code. Sounds good.
- 39:17- All right, let's do it.
- 39:18Alright, thanks as always for the time.
- 39:19- Yes, thank you. And we will talk to you again next week.
- 39:23- All right, perfect. Thanks Ben. Yep.
- 39:25- Bye-Bye Scott. If you enjoyed the podcast,
- 39:29go leave us a five star rating in iTunes.
- 39:32It helps to get the word out
- 39:33so more IT Pros can learn about Office 365 and Azure.
- 39:37If you have any questions you want us to address on the show
- 39:40or feedback about the show, feel free
- 39:42to reach out via our website, Twitter, or Facebook.
- 39:45Thanks again for listening and have a great day.