Latest / Microsoft Cloud IT Pro Podcast / Episode 386 – Exploring DevOps Tools: Maester, Pester, and SCUBA
Transcript
- 0:03Welcome to episode
- 0:05386
- 0:05of the Microsoft Cloud IT Pro Podcast recorded
- 0:09live on October 4, 2024.
- 0:11This is a show about Microsoft 365
- 0:14and Azure from the perspective of IT pros
- 0:16and end users, where we discuss the topic
- 0:18or recent news and how it relates to
- 0:20you. Today, we explore 3 powerful tools designed
- 0:24to improve automation, testing, and security, and modern
- 0:28and operations workflows with Microsoft 365.
- 0:32These tools are Maestro, Pester, and ScubaGear.
- 0:35Whether you're a DevOps engineer, PowerShell enthusiast, or
- 0:38security professional,
- 0:40this episode provides insights on how you can
- 0:42streamline your processes and strengthen your infrastructure
- 0:46using these powerful tools. Tune in to level
- 0:49up your automation and security game.
- 0:53What's my upload? 924
- 0:55with an upload
- 0:58of the same, 900.
- 1:00It is not my Internet. I can confirm
- 1:02that. When you left and came back, did
- 1:03I come back? Okay. Let me
- 1:06I don't know. Technology
- 1:08woes abound, Ben. Technology woes abound. Yes.
- 1:12Because back to my Teams issue, your face
- 1:14is now covered up by my audio panel
- 1:16where I select my microphone and my speaker
- 1:19because apparently,
- 1:20in my tenant, in my client, on my
- 1:22Mac,
- 1:23some combination
- 1:25of them, for whatever reason, if I minimize
- 1:28my audio
- 1:30selection
- 1:31pane
- 1:32and go click on another window, Teams loses
- 1:35my audio. Like, completely gone. Telling you, it's
- 1:37a feature.
- 1:39It explodes and disappears into thin air and
- 1:41Scott can no longer hear me. I come
- 1:43back to Teams, I click the little arrow,
- 1:45my audio panel drops up, and my audio
- 1:47magically returns.
- 1:49Ironically,
- 1:49with all of this, do you know how
- 1:51I knew that was an issue when you
- 1:52couldn't hear me? Besides me, like, waving my
- 1:54hands and going, is it you? Is it
- 1:56me? And yeah.
- 1:58Because my laptop,
- 1:59my MacBook Pro, and my Mac Studio
- 2:03both do the same thing,
- 2:05which means it is not an isolated issue.
- 2:08At least it's consistently bad. You got that
- 2:10going for you. I suppose.
- 2:12The other thing I have noticed though that
- 2:14seems to be consistent with it is it's
- 2:16when my audio interface is plugged in. So
- 2:18this may be a 3 pronged issue between
- 2:21Teams,
- 2:22a road Procaster 2 audio interface,
- 2:25and Mac
- 2:26OS because I was just on a meeting
- 2:28right before this and my speaker phone was
- 2:31working just fine. Although, Teams did crash once
- 2:33in the middle of main meeting and my
- 2:35Mac completely crashed and rebooted in the middle
- 2:37of the meeting. Can't necessarily attribute that to
- 2:39Teams. That being said,
- 2:42it was working fine on my USB speaker
- 2:44phone, and my audio interfaces seems to be
- 2:46the problem. It was very similar
- 2:49on my MacBook Pro where it was an
- 2:52audio interface that it didn't work with, and
- 2:54it seemed to work better if I use,
- 2:55like, the built in audio or AirPods.
- 2:58Either way, it's just weird, and I should
- 3:00probably submit some feedback, but I honestly have
- 3:02no idea how to record this to even
- 3:05submit feedback on how this is broken. I
- 3:08got no good ideas for you there other
- 3:10than call up your favorite engineer on the
- 3:13Teams team and show them.
- 3:15Let's get on a call. Try and do
- 3:17a call, but it's gonna be a rough
- 3:18one. Can we do it in Zoom so
- 3:20it just works and it's ready to
- 3:23go. But It's weird. I do have a
- 3:26ticket open for another Teams issue and that
- 3:29Teams freezes every time I try to switch
- 3:31audio devices.
- 3:32Could be related. End of the story, I
- 3:34just have some weird audio issues in Teams
- 3:36right now.
- 3:39Boy, we'll get it all sorted out, and
- 3:43we'll we'll figure it out.
- 3:44Anyhoo, yeah, why why don't we get going
- 3:46with our day today? Where should we go?
- 3:48To security and beyond with tools.
- 3:51We can go
- 3:53wherever you would like to go. Easy enough.
- 3:56Alright. So we've been talking about security. We'll
- 3:59stick with our plans, Scott. We have an
- 4:01agenda as long as my audio holds out
- 4:04for you.
- 4:05Our agenda is
- 4:07security tools because we have been talking about
- 4:10security in various
- 4:11forms,
- 4:12fashion,
- 4:13Azure,
- 4:14Microsoft
- 4:15365,
- 4:17Sentinel,
- 4:18reports,
- 4:19log analytics,
- 4:21all of those things. And truth be told,
- 4:23we started down this path because
- 4:26we had some questions
- 4:28from I don't know if it came from
- 4:30listeners, if it came from Discord, if it
- 4:31came from a combination of the 2, really
- 4:34about some of the third party tools that
- 4:36have
- 4:37arisen
- 4:38lately around
- 4:40I would say a lot of it's security
- 4:42testing
- 4:43and hardening,
- 4:44reporting,
- 4:45all of that against Microsoft Cloud Environments, and
- 4:49that led us down like a 4 week
- 4:51rabbit hole of just built in security. And
- 4:53we have finally arrived
- 4:55at a discussion around security tools.
- 4:59Here we are. We've made it. Which one
- 5:01would you like to start with? Would you
- 5:03like to take a pick? Take your pick.
- 5:05I think there's a bunch of different things
- 5:08out there that we could potentially
- 5:11talk about and
- 5:13take a look at. So when you're thinking
- 5:14maybe in context of
- 5:17what are the tools that can help me
- 5:19do my job, there's certainly things like, we
- 5:20talked about Sentinel last week, which arguably is
- 5:24both like a tool and a security service.
- 5:26You got that whole cloud based SIEM thing
- 5:28going on. Really, what we're talking about today
- 5:31is, like, tools that you can install on
- 5:33your client, and you can run them, and
- 5:35you can potentially manipulate them
- 5:38and and move them forward.
- 5:40That might be something like if anybody's ever
- 5:42heard of Bloodhound.
- 5:45There's things like Scuba, the secure cloud business
- 5:48application
- 5:50stuff.
- 5:51And there's also one from
- 5:53our buddy, Merrill, over on the Azure Cat
- 5:57team
- 5:58and the Entra I guess it's EntraID cat
- 6:01or EntraCat?
- 6:02Isn't it Microsoft Entra? Aren't we supposed to
- 6:05always preface it with Microsoft?
- 6:07I don't know. I can't keep up with
- 6:09these things. That CAT team. You keep me
- 6:10honest, Rory. That that that CAT team. So,
- 6:14Merrill has been busy on the side. He
- 6:15I think when we had him on the
- 6:16show
- 6:17a while back, he talked about a bunch
- 6:19of the tools he built, and this is
- 6:21another new tool that he built, which is
- 6:23called Meister.
- 6:25So why don't we
- 6:27start
- 6:28with Meister as our first
- 6:30command line, hey, you can pick this up
- 6:32and run with it kinda tool. There we
- 6:34go. I think I had an audio glitch,
- 6:36Scott, because I had to go do something.
- 6:39Yes. This is out there on GitHub.
- 6:42And if you're in Discord watching this, we
- 6:46can have some of it on the screen
- 6:47too, but we will place the show notes
- 6:50or place the link
- 6:52for Maestro in the show notes. This is
- 6:55a open source project.
- 6:57It's meister.devin.
- 6:59It's m a
- 7:00e s t
- 7:01e r is, like, the
- 7:04home page for it, and they have a
- 7:06lot of instructions, guidance here on how to
- 7:09get started. So I actually
- 7:12was
- 7:12testing this out and using it for
- 7:16a client today and went through, like, the
- 7:19whole process of setting it up and doing
- 7:22an initial run of it in 15 or
- 7:2420 minutes. So it uses
- 7:28Pester to do a bunch of tests. So
- 7:30if you go to the website and go
- 7:31to the docs and
- 7:33go to the installation guide,
- 7:36you install
- 7:38the Pester module in PowerShell,
- 7:41and then you install the Meister module. So
- 7:44this is out in the public PowerShell gallery,
- 7:47so you can do install module for both
- 7:49Pester and for Meister. You go create a
- 7:53directory to run your tests in, and then
- 7:55you do an install
- 7:57Meister tests that goes and
- 8:00pulls certain
- 8:01modules, certain content down into
- 8:04that directory.
- 8:06And then they even have all the connections
- 8:07in there. So you can go do connect
- 8:09to Meister, and it'll go connect to your
- 8:12environment,
- 8:13may prompt you for some permissions from app
- 8:15some app permissions. So I will say the
- 8:1815 to 20 minutes is, like, assuming you're
- 8:20a global admin, and you can just click
- 8:23to approve Next, your way through
- 8:25it. Next your way through, and then do
- 8:27an invoke Meister.
- 8:29And then it actually just goes and runs
- 8:30all the tests.
- 8:31They also have some optional stuff, so it
- 8:33includes some optional,
- 8:35CISA tests,
- 8:37and
- 8:38those are skipped if the connections and modules
- 8:41aren't there. So if you wanna get, like,
- 8:42the whole
- 8:44gamut of tests,
- 8:46you should also go install the AZ module.
- 8:49And I saw some stuff on here. Realistically,
- 8:52you don't need all of the AZ module,
- 8:54but it isn't defined in here what it
- 8:56needs. Some people are like, can we limit
- 8:58this so it doesn't take 10 minutes to
- 9:00install
- 9:01all of the modules that are part of
- 9:03AZ and limit it just to what we
- 9:04need? Probably go figure it out. And then
- 9:07the Exchange Online Management
- 9:09module because
- 9:11Exchange Online is where all the security and
- 9:14compliance
- 9:14PowerShell things are, which, again, fascinates me. That's
- 9:18still all bundled in the Exchange module and
- 9:20the connection to Exchange Online. But if you
- 9:23install those,
- 9:24you'll get some additional
- 9:26tests that require those additional permissions, those additional
- 9:30connections
- 9:31to the cloud. But then you just go
- 9:34connect to all of those. There's a I
- 9:37would say there's a couple different ones. There's
- 9:38a connect dash meister that just does, like,
- 9:41m 365 connection. If you install Exchange AZ,
- 9:45you can do a connect meister dash service
- 9:47all, and it goes and connects
- 9:50everything. Sorry. My wife just waved at me,
- 9:52and I don't know where she's going. She's
- 9:54leaving.
- 9:55I can't remember if she has Squirrel. Oh,
- 9:56I know where she's going. Okay. Yes. Absolutely,
- 9:59squirrel. Anyways, once you connect and invoke it,
- 10:01it goes and actually just uses PowerShell.
- 10:04And this is where
- 10:06we can get into this a little more
- 10:07scattered about customizing it, but it runs all
- 10:09those built in tests,
- 10:11against Exchange, against Azure, security compliance,
- 10:15spits them all out into that folder you
- 10:18created. So I should go look at in
- 10:20that folder that it creates,
- 10:23it creates like some HTML
- 10:25files,
- 10:27it creates
- 10:28JSON files, it creates some markdown files, but
- 10:31all of those spit out a really nice
- 10:34HTML file that gives you a summary of
- 10:37everything. So for instance, the environment I ran
- 10:40this against, it ran a 145
- 10:42different tests. It gives me a dashboard that
- 10:44says it passed 63 of them. It failed
- 10:4763 of them, and then there were 19
- 10:50of them that were not tested. I did
- 10:52have some issues connecting to Azure in this
- 10:55particular tenant, which might be some of the
- 10:56ones not tested. And then if you
- 10:59it gives you a couple of graphs that
- 11:01shows categories too around, like, at management policies,
- 11:05default settings,
- 11:07consent policy settings,
- 11:09and then it gives you a list of
- 11:11all of the different tests
- 11:14and even what policies against security frameworks
- 11:18these align with and
- 11:21if you passed it and you failed it.
- 11:23So this one, the first one was authentication
- 11:25math method, FIDO 2 security key state,
- 11:29and that one was passed. Something else further
- 11:32down was a FIDO 2 security key and
- 11:34force key restrictions.
- 11:35They don't have key restrictions enforced, so that
- 11:37one was failed. And it'll go through and
- 11:39look at Microsoft Authenticator settings, MFA settings,
- 11:43authentication
- 11:44methods that are enabled and disabled,
- 11:47admin consent,
- 11:48app requests,
- 11:50MFA.
- 11:51Those are some of the Azure AD ones,
- 11:53Exchange. It'll look at things like DMARC and
- 11:56DKIM, SPF on your domains,
- 11:59external warner
- 12:01external sender warnings, conditional access policies.
- 12:05All of those are some of the different
- 12:06tests that runs. The nice thing is next
- 12:09to each one by a pass or fail.
- 12:11It gives you a little view details where
- 12:13you can click in, and it'll tell you
- 12:16this one. Activation of global administrator role shall
- 12:19require approval. Your tenant has active assignments without
- 12:22a start date.
- 12:23So this one, there's some global admins that
- 12:25aren't pinned, they don't have an active start
- 12:28date, and it gives an explanation.
- 12:31Then it also gives
- 12:32remediation actions. So if you wanna go fix
- 12:35this, here's this one has 9 steps to
- 12:39go in and fix it, and then it
- 12:40also gives you related links to Microsoft learn
- 12:43document
- 12:44nope. This one goes to the enter admin
- 12:46center, then it gives you links
- 12:49to the,
- 12:50CISA I don't know. What is that? The
- 12:52number, the virtual number, the article number. This
- 12:56is 7.6,
- 12:57highly privileged user access. A link out to
- 13:00the CISA gov
- 13:02documentation around this and then
- 13:05some additional reference links as well. So it
- 13:06gives you a lot of information, not just
- 13:08about what you passed and failed, but how
- 13:10to remediate it and why and where
- 13:14this particular
- 13:15best practice or security
- 13:18guidance comes from.
- 13:19Yeah. So this is a really cool tool,
- 13:21and there's a lot of moving pieces, but
- 13:23all the, like, work has been done for
- 13:27you by
- 13:28Merrill and the others who have contributed to
- 13:30this project alongside them.
- 13:33You mentioned
- 13:34that
- 13:35Pester is a required dependency for this. So
- 13:38Pester,
- 13:39for those that haven't run into it, is
- 13:43a
- 13:44a I've traditionally used it as like a
- 13:46unit testing framework for PowerShell,
- 13:48but it's really like a mocking framework. So
- 13:51you can go do and potentially write unit
- 13:54tests for
- 13:55hey. I have a functions
- 13:57that
- 13:58I I don't know. I I wrote a
- 14:00function that returns the list of 50 US
- 14:02states, and you wanna make sure that function
- 14:05actually returns 50 states and not 49,
- 14:08not 51, not 52, things like that. So
- 14:11you could write a pester test against
- 14:14that given function,
- 14:16and it would tell you basically, hey, does
- 14:18this thing pass, fail, and what's going on
- 14:20there? So they're doing a very similar thing.
- 14:23They've just taken that unit testing mentality
- 14:26and
- 14:27applied it to the logic
- 14:29and
- 14:30known set of rules for valid or
- 14:34preferred configuration
- 14:36for all these items and things like M365
- 14:39services, like entry ID
- 14:42or for exchange.
- 14:44You could extend this out and write your
- 14:46own tests for,
- 14:48really, anything that could be managed in PowerShell
- 14:50or talk to over a REST API because
- 14:52you can always just do, like, an invoke
- 14:54web request
- 14:55on the PowerShell side of things and spin
- 14:58it up.
- 14:59So there's this deep set of tests that
- 15:02you've described, and they're all documented
- 15:05on the meister.dev
- 15:07site. So there's a section down at the
- 15:10very bottom of the docs for test overview,
- 15:12and you can go in there and actually
- 15:14look at the individual tests that are run
- 15:16across these various dimensions for basically, like, Meister
- 15:20based tests, the CISA tests, anything that coming
- 15:24is potentially coming out of scuba, things like
- 15:25that. And you can either run these tests
- 15:28as is, or you can take them and
- 15:30you can edit them because these are all
- 15:32just pester tests that are authored in PowerShell.
- 15:36So once you wrap your head around the
- 15:38way
- 15:40that pester tests are
- 15:42composed
- 15:43and how you describe a test and how
- 15:45you put it together,
- 15:46and a test can be a a bunch
- 15:48of different things in pester. It could be
- 15:51like,
- 15:52hey,
- 15:53the mock out of something.
- 15:55They can do you can do what if
- 15:57types of things happen.
- 15:59You can do things like, hey, should this
- 16:03be this way?
- 16:04So it has all these kind of descriptive
- 16:06words that you can write tests in context
- 16:09of that and and then go ahead and
- 16:10run them. And then the really cool thing
- 16:13is so this is all built for you.
- 16:14This is ready to go and you'd potentially
- 16:16have to rationalize,
- 16:17okay, what are the set of tests that
- 16:19I wanna run and things like that. So
- 16:22you just mentioned the outputs that come out
- 16:24of this,
- 16:25where we have things
- 16:27like
- 16:28that markdown file, that HTML file, so all
- 16:31that context from running the test that then
- 16:34become makes it actionable and and shows you
- 16:37the pass fail state of those tests.
- 16:39The way this thing is set up is
- 16:42because it's just PowerShell,
- 16:43you could run it from your laptop, like,
- 16:46you can run it from your Mac, you
- 16:48can run it from a Windows box. Like,
- 16:49it's gonna be in context of,
- 16:51can you install, like, the Exchange modules if
- 16:54you wanna do, like, the connectivity with the
- 16:56testing with Exchange, things like that. Yep. But
- 16:58because it can run from anywhere,
- 17:01the other great thing that they've done is
- 17:04they've put guidance out there
- 17:06for how to
- 17:08automate running these things. So if you're gonna
- 17:10go down the path of saying, hey, let
- 17:12me run an assessment across my against my
- 17:14environment,
- 17:14One of the general things that you would
- 17:16look for is running that assessment multiple times
- 17:18and then gauging where you go. Does my
- 17:21score improve? Does my pass rate improve? Things
- 17:24like that. So you can take all this
- 17:26and you can wire it up inside of,
- 17:29say, Azure DevOps
- 17:31and an ADO pipeline.
- 17:34You can run it inside of Azure Automation
- 17:37where PowerShell can be consumed. You could run
- 17:39this thing in a container
- 17:41and just sidecar it and have a container
- 17:43that spins up, spins down on a defined
- 17:46schedule, things like that. So you can really
- 17:48take this and treat it, 1, as a
- 17:51point in time, k. Give me a snapshot
- 17:53of my state today, but then you can
- 17:55also take it and automate it and operationalize
- 17:59the whole thing end to end if you
- 18:01want to.
- 18:02And it's all very
- 18:04consumable
- 18:05as far as, like, the reports that it
- 18:07puts out, so you had that HTML view
- 18:09up earlier. Yep. It's not like going into
- 18:11the rich Power BI report with a bunch
- 18:13of slices or things like that. It's basic
- 18:15HTML,
- 18:16but you could just have that running automatically
- 18:18in the background, say, like, once a week
- 18:21or once day, whatever your flavor is, and
- 18:24you could constantly
- 18:25be checking the output of that HTML.
- 18:27You can put your manager to it or
- 18:29your manager's manager, your boss's boss, and they're
- 18:32gonna be able to figure out what's going
- 18:33on and see the big blocks for oh,
- 18:36I had
- 18:377 passes yesterday
- 18:39and 10 failures,
- 18:41And
- 18:42today, I've got 10 passes
- 18:44and 7 failures, so we're incrementally improving.
- 18:47So I I think from that perspective,
- 18:50super powerful.
- 18:52Maryland team have done all the heavy lifting
- 18:55around
- 18:56translating best practices across these workloads
- 18:59from both the lens of Microsoft
- 19:01and from external entities like CISA. And they've
- 19:05already written all the mocks and all the
- 19:07tests out there, and then you can go
- 19:08in ahead and extend to your heart's content.
- 19:11Right? So if you don't want that
- 19:13global admin test run, great. Delete it or
- 19:15just don't run it. You want to change
- 19:17the logic of that test so that logic
- 19:20maybe ignores a couple of your break glass
- 19:22accounts or things like that, great. Go ahead
- 19:25and change it. It's just a PowerShell unit
- 19:28test written in Pester. So as long as
- 19:30you're adhering to the Pester framework, it's all
- 19:32very straightforward.
- 19:34It's super slick, super turnkey. Like, I would
- 19:37recommend, like, folks, like, even if you're just,
- 19:39like, a PowerShell geek and you've never done
- 19:41Pester,
- 19:42this is a good introduction to Pester as
- 19:44well, like, without having to go and do
- 19:46a bunch of other weird stuff on the
- 19:48side.
- 19:52Do you feel overwhelmed by trying to manage
- 19:54your Office 365
- 19:55environment? Are you facing unexpected issues that disrupt
- 19:58your company's productivity? Intelligink is here to help.
- 20:01Much like you take your car to the
- 20:02mechanic that has specialized knowledge on how to
- 20:05best keep your car running,
- 20:07Intelligent helps you with your Microsoft cloud environment,
- 20:09because that's their expertise.
- 20:11Intelligent keeps up with the latest updates in
- 20:14the Microsoft cloud to help keep your business
- 20:16running smoothly and ahead of the curve. Whether
- 20:18you are a small organization with just a
- 20:20few users up to an organization of several
- 20:22thousand employees,
- 20:24they want to partner with you to implement
- 20:26and administer your Microsoft Cloud technology.
- 20:29Visit them at inteligink.com/podcast.
- 20:33That's intell
- 20:37ing.com/podcast
- 20:40for more information or to schedule a 30
- 20:42minute call to get started with them today.
- 20:45Remember, IntelliJunk focuses on the Microsoft cloud, so
- 20:48you can focus on your business.
- 20:52So I've started doing this for a couple
- 20:54clients now of mine, and that's where I've
- 20:56started even diving into this more as they're
- 20:59like, hey, Ben. We wanna do, like, weekly
- 21:02check ins with you, where you go set
- 21:04this up, you run it, you review the
- 21:06reports, we meet, we discuss
- 21:09what failed, what passed, do we wanna do
- 21:11anything about it, do we want not wanna
- 21:13do anything about it. And I had a
- 21:15call with one of my clients today about
- 21:17it where we looked at this and it's
- 21:19absolutely our plan is to go in. We're
- 21:21gonna set this up maybe in Azure Automation,
- 21:24maybe Azure DevOps.
- 21:25So this is running on a regular, probably
- 21:28weekly basis, and there were some of these.
- 21:30We're like, like, giving your scenario, how your
- 21:33company works, it's showing failed, but we don't
- 21:36necessarily
- 21:37want this one to be failed because you
- 21:39have a valid reason for it to be
- 21:41set the way it is. In some of
- 21:43them, we actually found where they were,
- 21:46I would say the recommendation
- 21:48is actually to be a little bit more
- 21:49open than what they were. There are certain
- 21:51things that they actually just locked completely down.
- 21:53They turned it off,
- 21:55and the best practice is to not actually
- 21:57have it off, but to have it where,
- 21:58like, users could request access or it goes
- 22:01through a workflow that's
- 22:02technically, we passed it because users, they don't
- 22:05even have to go through a request because
- 22:07it's just completely turned off. But going in
- 22:09and starting to customize this, tweak it, and
- 22:12building from that a list of here's what
- 22:14we should do in your environment to
- 22:17adhere to best practices,
- 22:18to make sure it's secure. And like you
- 22:20said, you can do it they have guides
- 22:22in here, fresh automation, DevOps, and then they
- 22:24even have one in here for email alerts.
- 22:27So if you want to email this report
- 22:29out regularly,
- 22:30they have Slack alerts. Apparently, we can do
- 22:32email alerts in Slack alerts, but we can't
- 22:34do Teams alerts. Maybe that goes back to
- 22:36our conversation earlier and yeah. We should just
- 22:39have easy webhooks and we don't. Yep. I'm
- 22:42saying, yeah. But then even some custom tests.
- 22:44You could.
- 22:45Like, you can
- 22:47at the end of the day, like, this
- 22:49is a framework that's been prebuilt for you.
- 22:51They said you just go download it, leverage
- 22:53it. But then, yeah, because it's all just
- 22:56a bunch of PowerShell
- 22:57files. Right? It's just text files. Like, you
- 22:59can go see it all, you can manipulate
- 23:02it, and you can turn it into your
- 23:06own needs.
- 23:07So where this thing potentially focuses
- 23:10on,
- 23:11like, that that Microsoft 365
- 23:14stack and the things that are going on
- 23:15there, I think this also fits very nicely
- 23:17into the world of Azure where maybe you
- 23:19wanna extend it and you wanna write some
- 23:21unit tests around
- 23:23configuration
- 23:24of your management groups and application of policy
- 23:28within those management groups. You want to
- 23:31create something about, like, resource governance, an application
- 23:35of
- 23:36the right roles or a known set of
- 23:38roles in identity and access management. You want
- 23:40to do, like, a policy test kind of
- 23:42thing.
- 23:43It's all there. It's just sitting there waiting
- 23:45for you to pick it up. And like
- 23:47I said, as long as you can write
- 23:48a PowerShell script, you're off to the races.
- 23:52Yep. Super cool, super
- 23:54powerful stuff. I've played with Pester in the
- 23:57past, and I've used it here and there.
- 23:59I never would have thought to use it
- 24:00for something like this.
- 24:02And it's just it makes perfect sense once
- 24:04you think about it. Oh, yeah. This is
- 24:05a natural fit. I never would have thought
- 24:07about it myself, and it's super cool that
- 24:09Merrill and team did the work to bring
- 24:11this out into the world. So when you
- 24:12wanna go help me write some custom tests,
- 24:14Scott, I already have a list started of
- 24:16custom tests I wanna write for
- 24:18my client or like you said, for Azure,
- 24:20it would be super cool to start writing
- 24:22some extending this to some of those tests.
- 24:25It's super
- 24:26quick to
- 24:28pick up.
- 24:30Again, these are all just pesters. So as
- 24:32long as you understand, like, the keywords for
- 24:33pester,
- 24:35you're not writing maybe, like, a function in
- 24:37PowerShell, but you'll do
- 24:39you'll describe an action and describe a context
- 24:42kinda thing.
- 24:44You can do that. And then they've structured
- 24:45it in a way where,
- 24:47they follow pester best practices
- 24:50where so every pester test
- 24:53ends with a suffix, like it's just a
- 24:55PowerShell file like a PS one script,
- 24:58but it's always
- 24:59something dot tests dotpsone.
- 25:01So when you install Meister, there's going to
- 25:04be,
- 25:06a folder there for your custom tests. You
- 25:09can just dump those PowerShell scripts in there,
- 25:12And as long as they follow
- 25:14the
- 25:16the syntax and and what Pester,
- 25:18expects, right, like that like dot test dot
- 25:21psone suffix,
- 25:22those will go ahead and run on your
- 25:24next run automatically.
- 25:25Yeah. Even looking to your point about it
- 25:28being super simple, looking at their guide on
- 25:30how to write custom tests, the fact that
- 25:33the documentation
- 25:34can be this short for adding a custom
- 25:36test to a file,
- 25:37I get based on what you're testing for,
- 25:39you may have to write more PowerShell, but
- 25:41this is super straight forward to go in
- 25:43and start implementing your own stuff. The other
- 25:45thing that's is super cool in here is
- 25:48there's a bunch of conditional access,
- 25:51what off What if. What if tests. I
- 25:53don't know if you had a chance to
- 25:55look at any of these
- 25:58and how they compose together, but you can
- 26:01do things like
- 26:02do, like, conditional what if statements against
- 26:06would this user ID be impacted by a
- 26:09given policy that you've implemented
- 26:12or things like that. It's, again, just super
- 26:15turnkey
- 26:16and super powerful at the same time. Like,
- 26:19I I I love stuff like this, and
- 26:21it's all open source, like, it's free. I
- 26:23I I love that you're picking it up
- 26:25and running and and taking it to
- 26:28customers and extending it out that way. That
- 26:30just shows, like, how kinda turnkey it can
- 26:32be. I wonder if you could do these
- 26:34conditional ones. I was looking at the conditional
- 26:36what if tests
- 26:38should contain
- 26:39and I'm guessing there is a way it
- 26:41queries that, runs the
- 26:43that blocks Azure. So this is testing if
- 26:45there's the access there. If you could somehow
- 26:48test where if you have certain what if
- 26:50tests, if a user is connecting from a
- 26:53certain IP address with a certain risk level
- 26:55to and maybe this is doing this and
- 26:57I'm not reading it quite right, where in
- 26:59that report that you get weekly,
- 27:02did conditional access get changed in a way
- 27:05that your what if test
- 27:07starts to
- 27:08essentially, a contextual access test for a certain
- 27:11scenario becomes invalid because someone made a change
- 27:14to it. Would this show a fail in
- 27:16there where all of a sudden
- 27:18these users aren't getting prompted for MFA?
- 27:21Do you know what I'm saying? Where you're
- 27:23actually testing what ifs as a part of
- 27:24your weekly run. That's actually what this is
- 27:26doing. Is that what this is doing? Okay.
- 27:28If you look under the hood, so this
- 27:30is using
- 27:31the
- 27:32test MT conditional access what if commandlet. Yep.
- 27:36And that cmdlet is not part of
- 27:39Meister. That cmdlet is part of the official
- 27:42tooling that Microsoft gives you for actually it's
- 27:46like the official what if tool to troubleshoot
- 27:48conditional access policies
- 27:51from Microsoft
- 27:53themselves.
- 27:54So it's just
- 27:56it's using the same underlying things, same set
- 27:58of rest APIs, all that kind of stuff.
- 28:01So I should have known that. Yeah. This
- 28:03is super easy.
- 28:05So if you've ever done what if tests
- 28:07for conditional access inside of, like, the native
- 28:11portal experience, then that that's effectively
- 28:14Yeah. So so that's effectively what you're doing
- 28:16is you're just running those same tests. You're
- 28:18just mocking them inside of PowerShell. Got it.
- 28:20And then you're looking for
- 28:23in Meister,
- 28:24when that test runs,
- 28:27what is the results of that test? Is
- 28:30it a block? Is it a fail? Is
- 28:32it ignored? Is it not applied, etcetera? Joshua
- 28:35Sharfstein: You're basically looking for
- 28:38a truefalse to come out of the what
- 28:40if, right? What was the
- 28:42the expected outcome was pass
- 28:44or pass fail kind of thing? Yeah. Matthew
- 28:46Bunnieski: Nifty. Maestro, absolutely something you should go
- 28:49check out. The other one, let's do a
- 28:51little bit of a comparison, Scott. We have
- 28:53a few minutes
- 28:55ish sort of 5 minutes.
- 28:575 minutes? Yeah. Let's do it. Let's push
- 28:59it. Right push it in it. So this
- 29:01is another one is
- 29:03scuba gear. I'm gonna go pull this website
- 29:05up. Oh, seriously? Let me tell you what
- 29:07I think about Bastion lately. It's right up
- 29:09there with Teams for me. Scuba gear is
- 29:12another one that is out there that I
- 29:14would say is very similar
- 29:16to
- 29:17to Meister, only this one comes straight from
- 29:21CISA. So they wrote their own
- 29:24cybersecurity
- 29:27testing
- 29:28PowerShell
- 29:29module that is also open source out there
- 29:32in GitHub, and it's an assessment tool for
- 29:34Microsoft 365 tenant configuration
- 29:37conforming
- 29:38to the scuba,
- 29:40which is the security cloud business application
- 29:43baselines.
- 29:44And this one is very similar
- 29:48to Meister and how you set it up
- 29:49that there's a scuba gear module out there
- 29:51in the PowerShell gallery. So you go install
- 29:54module scuba gear, you initialize it, which goes
- 29:57and downloads all the dependencies.
- 29:59You can go run, see which version is
- 30:01included,
- 30:02and then you invoke scuba gear with the
- 30:04product names. And I just passed in star,
- 30:07but you can pass in various product names
- 30:09and this one is slightly different
- 30:12in that it does include
- 30:14I'm just gonna go to it, Scott. Nobody's
- 30:16looking at this. It includes I have my
- 30:18tenant name and ID displayed in here. I
- 30:20was gonna keep it in so nobody knew,
- 30:22but since nobody's watching it. It includes products,
- 30:24so this one
- 30:25reaches a little bit further than Meister does
- 30:28in some of the tests where it looks
- 30:29at it still says Azure Active Directory,
- 30:32enter, but then it looks at Microsoft 365
- 30:34Defender, it looks at Exchange, it looks at
- 30:37the Power Platform, SharePoint, and Teams,
- 30:40and does something similar where it runs tests
- 30:42and then gives you passed, warnings, failed.
- 30:46And this one also says manual checks needed
- 30:49where it couldn't automatically
- 30:51determine
- 30:52a certain condition,
- 30:54and it'll go in and tell you
- 30:56how to manually check for a certain
- 30:59security
- 31:00setting. So same type of thing, spits out
- 31:02a bunch of HTML
- 31:05markdown,
- 31:06etcetera,
- 31:07to your computer.
- 31:08So you could also set this up to
- 31:10run-in a DevOps pipeline or something else. Once
- 31:13you click on a certain
- 31:15category,
- 31:15just pick power platform,
- 31:17it goes in and says, here's the control
- 31:20ID for power platform.
- 31:22What do you want? The ability to create
- 31:23production in sandbox
- 31:25environment shall be restricted to admins. Trial environments
- 31:28are restricted to admins.
- 31:30DLP policy to restrict connector access.
- 31:33NondeFAULT
- 31:34environments should have at least one DLP
- 31:36policy affecting them. Allow inbound, outbound, connection allow
- 31:41list should be configured,
- 31:43content security policies. So it gives you the
- 31:45same type of thing where a control ID,
- 31:48what the requirement is,
- 31:50what the result is, a criticality,
- 31:53should be done,
- 31:54shall be done, so recommendation versus
- 31:57requirement,
- 31:58and then a few details around
- 32:01I would say not as many
- 32:03as Maestro does. Maestro gives you, like, step
- 32:05by step. Some of these say, under details,
- 32:08the requirement is not met. So you're gonna
- 32:10be on your own to go figure out
- 32:13what do you actually have to do in
- 32:15this case to go in and configure it.
- 32:17Other ones do have,
- 32:20like, the manual check ones. Usually, in the
- 32:22details, say it doesn't have the capability to
- 32:24check. Here's some instructions on how to do
- 32:28the manual check. This one, it also breaks
- 32:30it up into a bunch of different HTML
- 32:32files. So, like, Power Platform has its own,
- 32:35SharePoint has its own,
- 32:37Entra has its own, Defender has its own.
- 32:39And when you get into some of the
- 32:41things like,
- 32:43Entra specifically,
- 32:45there's gonna be some redundancy there. What I
- 32:47have thought would be interesting is to actually
- 32:49see if you could somehow take some of
- 32:50these tests that they have in scuba gear
- 32:53since that's all open source and take Meister
- 32:55since that's all open source and somehow combine
- 32:57them to
- 32:59include maybe some of these additional SharePoint teams
- 33:02power platform checks into the Meister checks so
- 33:05I could just have one tool that gives
- 33:07me everything? Yeah. You could. So
- 33:09Meister
- 33:11has a
- 33:12subset
- 33:13of the CISA tests in there. Yep. And
- 33:15those tests are actually coming out of
- 33:19CISA control IDs, which then all map back
- 33:21into
- 33:22the scuba project.
- 33:24So you're looking at effectively, like, that subset
- 33:27for SharePoint online, Exchange online, things like that
- 33:31that come out of CISA. But if you
- 33:33want, the nice thing that the Meister folks
- 33:36have done, if you go read their documentation,
- 33:39so if you hop into the CISA section
- 33:42for the Meister docs, they will tell you
- 33:46what they have implemented
- 33:47and what they haven't implemented.
- 33:49And in some cases, like, they'll give you
- 33:51the reason why they haven't turned it on.
- 33:55And for a lot of this, it's because
- 33:57they're focused
- 33:58on
- 33:59native tooling and kinda what's available to you.
- 34:02So scuba might go out and use, like,
- 34:04a nonstandard
- 34:05way to test for something within a given
- 34:08service, say, like, SharePoint online,
- 34:10And
- 34:11just knowing
- 34:12and having chatted with Merrill in the past
- 34:14and things like that, I bet one of
- 34:15the, like, the guiding principles here is, hey,
- 34:17this stuff just needs to be, like, in
- 34:19the graph and ready to go and easily
- 34:21retrievable. And if it's not, then we'd be
- 34:23doing it in a nonstandard way, and we
- 34:24don't really wanna show folks how to do
- 34:25it in a nonstandard way. I'm sympathetic to
- 34:28that. So if you go read the Maestro
- 34:29recommendation,
- 34:30so you have the page now for system
- 34:32controls for Microsoft SharePoint online. It'll say, hey,
- 34:35here's all the control IDs, and it'll just
- 34:38straight up tell you, like, oh, this one
- 34:40isn't implemented. And for any of them, you
- 34:42can just click the control ID, and it'll
- 34:44take you over to the scuba GitHub site,
- 34:47and you're just landed into that markdown file.
- 34:49So you could see, like, what the test
- 34:51was going to test for
- 34:53and how it was gonna come out. And
- 34:54then if you wanted to implement said test,
- 34:56yeah, you can absolutely do that. But it
- 34:59would be on you to implement it at
- 35:00that point. If you really want the world
- 35:02to be your oyster and have it all
- 35:03in one place,
- 35:04potentially some more work to do there. I
- 35:06I would bet that the
- 35:08Merrill and the folks who did Meister
- 35:10wouldn't mind if you just wrote some stuff
- 35:12up for him and contributed back. It's all
- 35:14also just on GitHub. Right? This is OSS,
- 35:17so you can go put a PR in
- 35:19if you want for a new test, or
- 35:21if you write a a new, like, really
- 35:23cool custom test or something like that and
- 35:25wanna share it with that community,
- 35:27you could absolutely do that through GitHub and
- 35:29things like that. Yeah. I may have to
- 35:30do that. Maybe I have to get involved
- 35:32in writing some new tests for Maestro and
- 35:35playing with some of that, submitting some of
- 35:36that up there because
- 35:38I agree. If I'm gonna do this, I
- 35:39might as well submit it so everybody else
- 35:41can take advantage of it as well. If
- 35:43you are an m 365
- 35:45admin,
- 35:46arguably
- 35:49an Azure admin, like, you're dependent on Azure
- 35:51Active Directory, I would totally give this one
- 35:54a spin.
- 35:55Spin it up, see what it does. It
- 35:56doesn't take you long to do. You should
- 35:58be able to carve out an hour and
- 35:59do this end to end Yep. Especially if
- 36:01you're running in, like, deity mode or you
- 36:03have the ability to elevate yourself into, like,
- 36:05global admin or something for a limited amount
- 36:07of time just to make your life a
- 36:08little bit easier for that first run to
- 36:10see, like, where you really stand in the
- 36:12world,
- 36:12and then you can just go from there.
- 36:14Awesome. Thanks, Scott. I'm now 5 minutes late
- 36:16for my next meeting, but
- 36:18it will be okay.
- 36:20Worth it. We will survive. Yes.
- 36:23So maybe we'll have some updates later with
- 36:25changes we've made, fill you in on what
- 36:27Maestro tests I've gotten written lately.
- 36:30But I'm with you. Like, these tools, the
- 36:33work Merrill has done on this is
- 36:35and others, we should say. It is not
- 36:37just Merrill. He has collaborated with us on
- 36:39a few others as well. So he is
- 36:41just the one that we first heard about
- 36:43it from, but they have done an outstanding
- 36:45job on this this platform, this framework, this
- 36:49tool, so absolutely go check it out. Yeah.
- 36:52I highly recommend it. Alright. Well, that's Scott.
- 36:54Enjoy your weekend.
- 36:56Don't work too hard. The weather is actually
- 36:58starting to be nice out. Maybe go outside,
- 37:00enjoy some
- 37:02weather in the eighties. Yeah. I was gonna
- 37:04say, I was outside the other day, and
- 37:05I was like, it actually feels decent out.
- 37:07And I got in the car, and I'm
- 37:08like, it's still 85. I guess that means
- 37:10I'm getting used to Florida.
- 37:13Only took a couple decades, but you're making
- 37:14it. Yeah. I'm getting there eventually.
- 37:16Alright. Sounds good. Thanks, Ben. Thanks, Scott. We'll
- 37:19talk to you later.
- 37:21If you enjoyed the podcast, go leave us
- 37:23a 5 star rating in iTunes. It helps
- 37:26to get the word out so more IT
- 37:27pros can learn about Office 365 and Azure.
- 37:31If you have any questions you want us
- 37:32to address on the show or feedback about
- 37:35the show, feel free to reach out via
- 37:37our website, Twitter, or Facebook.
- 37:39Thanks again for listening, and have a great
- 37:41day.