How I Test Conditional Access for CMMC Level 2 in Report-Only Mode

A bad Conditional Access policy never fails at a convenient time. It breaks sign-ins during payroll, blocks a sales laptop before a customer call, or leaves access to CUI exposed longer than it should.

In a CMMC Level 2 environment, I use Conditional Access in report-only mode to test first and enforce later. That gives me room to validate scope, tune exclusions, and collect proof without stopping work.

The line that matters most is simple: report-only is for validation, not enforcement. Once I keep that clear, the rest of the rollout gets much easier.

Why report-only matters in a CMMC Level 2 environment

For CMMC Level 2, identity controls are not paperwork. They affect who can reach systems that store, process, or transmit CUI. When I build a CMMC Conditional Access baseline, I want proof that the rule is aimed correctly before I turn it on.

That is where report-only mode earns its place. As Microsoft’s report-only guidance explains, it lets me evaluate the likely effect of most policies before I enforce them. I can see whether a sign-in would have required MFA, would have been blocked, or would not have matched the policy at all.

Report-only mode proves that I tested a policy. It does not prove the policy protected production access.

That distinction matters during assessment prep. An assessor can accept report-only evidence as part of control validation and tuning, but it is not the same as a production control that is set to On. If I claim enforced protection when the policy is still in report-only, I create both compliance risk and trust problems.

I also use this approach because outages are expensive. If a policy would break a field technician’s laptop or a finance team’s access to SharePoint, I want to find that out during a controlled test, not on a Monday morning. The same holds for admin accounts, remote workers, contractors, and line-of-business apps that still have odd authentication behavior.

Report-only also helps me separate design errors from enforcement errors. If the wrong users are in scope, if named locations are incomplete, or if a cloud app assignment is too broad, I can fix the design before the policy has business impact.

I plan the test before I build the policy

The cleanest report-only test starts before I open the Microsoft Entra admin center. I write down the control goal, the affected people and apps, and the evidence I expect to collect. That short prep step saves hours later.

My planning checklist is simple:

1. I state the control objective in plain language, such as “require MFA for all users accessing Microsoft 365” or “block legacy authentication.”
2. I map the policy to the CMMC practice or internal control statement it supports.
3. I name the user groups, cloud apps, device states, locations, and exclusions in scope.
4. I pick a pilot group that reflects real work, not only IT staff.
5. I protect emergency access by excluding break-glass accounts from the test.
6. I set a review window, an owner, and a go or no-go date for enforcement.

I follow Microsoft’s deployment planning guidance, which advises running new policies in report-only for at least a week before I enforce them. A week is often the minimum. If the policy affects contractors, mobile devices, or seasonal workflows, I run longer.

I use the same discipline across Small Business IT projects. During Cloud Infrastructure work, an Office 365 Migration, or a Data Center Technology refresh, one identity mistake can spread fast. The same is true in Restaurant POS Support and Kitchen Technology Solutions, where a blocked sign-in can stop orders and billing.

That is why I tie access policy work to Cybersecurity Services, Endpoint Security, Device Hardening, and Cloud Management. A solid Business Technology Partner brings Technology Consulting, Infrastructure Optimization, Digital Transformation, IT Strategy for SMBs, Secure Cloud Architecture, Managed IT for Small Business, and Business Continuity & Security into one plan. Those are the kinds of Tailored Technology Services and Innovative IT Solutions that keep security work aligned with real operations.

Before I move on, I also decide what “good” looks like. For one policy, success may mean every pilot user would have been prompted for MFA. For another, it may mean only unmanaged devices would have failed. If I cannot define success, I am not ready to test.

How I configure report-only policies in Microsoft Entra ID

Inside the Microsoft Entra admin center, I build Conditional Access policies as if they were going live, then I set the policy state to Report-only. I do not create a watered-down test policy unless I have a solid reason. The point is to validate the real design.

I start with naming. A name like “Policy 4” is useless three months later. I prefer a clear pattern such as “CA-Users-M365-RequireMFA-RO-CHG1042.” The name tells me the audience, target, control, state, and change record.

Next, I scope the policy carefully. I set users and groups, include the right cloud apps, and add only the exclusions I can defend. Every exception should have an owner and a date for re-check. If a service account cannot meet the rule today, I document the reason and plan its replacement.

Then I set the conditions and grant controls. Common CMMC-focused tests include requiring MFA, blocking legacy authentication, requiring compliant or hybrid-joined devices, and using stronger access for admins. In 2026, I often test authentication strengths for privileged roles so I can separate standard MFA from phishing-resistant access.

I also use the What If tool before I trust the logs. It helps me simulate a user, app, device platform, and location combination. Still, I treat What If as a preview, not final proof. Real sign-in data always carries more weight.

Break-glass accounts stay outside the policy. I keep those accounts cloud-only, tightly protected, and unused for daily work. If I ever lock out the tenant, those exclusions are what let me recover.

For device-based controls, I slow down. Report-only is useful for “require compliant device” testing, but I pay extra attention to macOS, iOS, and Android because prompts and client behavior can differ from what the policy preview suggests. Before enforcement, I run controlled live tests with real pilot users on those platforms.

The test scenarios I run and the logs I trust

A report-only test needs realistic traffic. If I only test with one Windows admin on a trusted network, I learn almost nothing. I want daily work, remote access, mobile use, and edge cases that might break under pressure.

These are the scenarios I usually test first:

Scenario	Policy under test	What I expect in report-only	Evidence I capture
Standard user signs in to Exchange Online and SharePoint from a managed Windows device	Require MFA for Microsoft 365	Log shows the policy would have required MFA or would have succeeded because MFA was already satisfied	Sign-in log details, policy result, pilot user note
User signs in from an unmanaged personal laptop	Require compliant or hybrid-joined device	Log shows the device-based policy would have failed or blocked access	Sign-in log, device state, screenshot of policy assignments
Admin accesses Azure or Entra admin portals	Stronger admin policy with authentication strength	Log shows the privileged policy would have applied to the admin account	Sign-in log, group membership, What If screenshot
Legacy mail client tries POP or IMAP auth	Block legacy authentication	Log shows the policy would have blocked the attempt	Sign-in record, client app detail, remediation note

After the sign-ins occur, I go to the sign-in logs and open the Conditional Access details for each event. I want to see whether the policy matched, what result it recorded, and why. A good result is not always “success.” Sometimes the useful finding is that a policy did not apply because a location, app, or group assignment is wrong.

When I need trend data, I use the Conditional Access insights and reporting workbook. It helps me spot repeated failures by app, user, platform, or location much faster than checking one sign-in at a time.

I also keep the limits in view. Report-only does not block the session. It does not force the user to finish MFA. It does not prove that a noncompliant device was denied in real time. For mobile and Apple platforms, and for app protection or device compliance paths, I treat report-only as directional. If a control is high-impact, I follow the report-only phase with a tightly managed live pilot before broad rollout.

The evidence I keep for CMMC reviews and change control

Good testing is not enough if I cannot show what happened. For CMMC work, I build an evidence package that another person can follow without guessing. If the record is weak, the test might as well not have happened.

My evidence set usually includes the policy screenshot, the full assignment scope, the policy state, the date range of testing, the pilot group membership, and sample sign-in results. I also keep the related change ticket, approval record, and remediation notes for any issues I found. If I change the policy during testing, I capture the before and after.

I write a short narrative beside the screenshots. The narrative says what I tested, why it matters, what the expected outcome was, what the logs showed, and what I changed before enforcement. That short note helps compliance leaders, MSP or MSSP operators, and Microsoft 365 admins read from the same page.

I do not rely on screenshots alone. Sign-in logs roll off, and retention depends on how the tenant is configured. Because of that, I export what matters while the evidence is fresh. A CSV, PDF, or controlled screenshot set is often enough if it is labeled well and tied to a change record.

For assessor readiness, I also link the evidence to the control family or internal standard it supports. I keep that language plain. “This policy was tested in report-only from May 3 to May 10. It would have required MFA for pilot users accessing Microsoft 365. After testing, I corrected two exclusions and moved the policy to On on May 14.” That kind of record is easy to defend.

If an MSP or MSSP runs the test for a client, I still want the client owner named in the evidence. Shared responsibility needs a name, not a vague handoff.

Mistakes that turn a safe test into risk

The first mistake is leaving report-only in place too long. A policy that sits in validation mode for months creates a false sense of protection. If the rule is important enough to build, it deserves a decision date.

Another common error is testing only friendly cases. IT staff on healthy Windows devices will usually pass. Contractors, executives, warehouse tablets, shared kiosks, BYOD phones, and older mail clients are where surprises live. If those user types exist in your scope, they belong in the pilot.

I also see teams combine too many changes at once. Requiring MFA, compliant devices, and stricter session controls in one move makes log review messy. When possible, I split controls into logical phases so I can see what caused each result.

Poor exclusion handling creates risk on both sides. Too many exclusions hollow out the control. Too few can lock out admins or business owners. I keep break-glass accounts excluded, but I review every other exception with a hard eye.

The What If tool can also mislead people when they treat it like final proof. It is useful, but it does not replace actual sign-in events. The sign-in logs tell me what happened with a real user, a real app, and a real device.

Last, weak documentation hurts good engineering. If I cannot show the pilot scope, review period, findings, and approval trail, the work becomes hard to defend during a CMMC review or a customer security questionnaire.

Final thoughts

When I test Conditional Access for CMMC Level 2, report-only mode is my rehearsal space. It lets me verify scope, catch surprises, and build evidence before a policy can affect production users.

The strongest takeaway is simple: report-only is a validation stage, not a control state. Once the logs are clean, the exceptions are justified, and the evidence is complete, then I move the policy to On with confidence.

That discipline protects compliance work, user access, and the business at the same time.