SharePoint Online Auth Failures - Modern Auth Backups Likely Impacted


Userlevel 7
Badge +20

Hi, Microsoft have a service alert for SharePoint Online.

 

If you’re doing everything right, odds are you’re using modern auth for your Microsoft 365 backups, in which case, you’re using the Graph API.

 

In this scenario, you may find you’re getting “Access Denied” errors in your backups, or even in your BAU processes.

 

If so you want to check out SP381039.

 

The latest information is some authentication components are missing for certain environments causing the failure. Microsoft are redeploying these components, no ETA as yet.

 

Full output here:

—-

Published Time: 17/05/2022 16:40:16


Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We've confirmed that a recent feature deployment misconfiguration has prevented components associated with the authentication feature from being available in a group of customer environments, which is producing 'Access Denied' errors when using Graph APIs for SharePoint Online. We've confirmed that our redeployment of the authentication feature to some impacted environments has resolved the impact. We're now redeploying the feature to all affected remaining environments, which is expected to remediate impact.

Scope of impact: This issue may potentially affect any of your users attempting to utilize Graph APIs for SharePoint Online.

Root cause: A recent feature deployment misconfiguration has prevented an authentication feature from being available in a group of customer environments, resulting in impact.

Next update by: Tuesday, May 17, 2022, at 9:30 PM UTC

Published Time: 17/05/2022 11:26:17

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We've identified that components of the authentication feature are unexpectedly not present in some users' environments thus resulting in the Graph API access requests to fail. We're redeploying the affected feature within impacted environments to remediate impact. In parallel, we're investigating recent feature changes to identity why the components are unexpectedly not present.

Scope of impact: Your organization is affected by this event, and any user utilizing Graph APIs for SharePoint Online may be impacted.

Next update by: Tuesday, May 17, 2022, at 5:00 PM UTC

Published Time: 17/05/2022 09:22:50

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We've identified that a specific feature, responsible for authenticating and authorizing user requests, is not performing as expected and causing some access Graph API access requests to fail. We're reviewing the affected feature to develop a remediation plan.

Scope of impact: Your organization is affected by this event, and the issue may impact any user utilizing Graph APIs for SharePoint Online.

Next update by: Tuesday, May 17, 2022, at 10:30 AM UTC

Published Time: 17/05/2022 07:28:14

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We reviewed the logs and identified a potential authentication issue with the Graph APIs. We're continuing our analysis of the authentication requests to determine our next troubleshooting steps.

Scope of impact: Your organization is affected by this event, and the issue may impact any user utilizing Graph APIs for SharePoint Online.

Next update by: Tuesday, May 17, 2022, at 8:30 AM UTC

Published Time: 17/05/2022 05:58:10

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We're analyzing system logs to help identify the source of the 'Access Denied' errors.

Scope of impact: Your organization is affected by this event, and the issue may impact any user utilizing Graph APIs for SharePoint Online.

Next update by: Tuesday, May 17, 2022, at 6:30 AM UTC

Published Time: 17/05/2022 05:31:39

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.


7 comments

Userlevel 7
Badge +13

Ups ...

Userlevel 7
Badge +17

Uii, this is serious…. 😱

Userlevel 7
Badge +20

Published Time: 18/05/2022 01:56:17


Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We've determined that a code regression is delaying the recovery in the remaining environments. We're reverting this code regression and expect the remaining environments to be fully recovered by our next scheduled update.

Scope of impact: This issue may potentially affect any user attempting to utilize Graph APIs for SharePoint Online.

Estimated time to resolve: We expect this issue to be completely resolved for all affected users within the next 15 hours as the remediation actions are safely implemented through our change management process.

Root cause: A recent feature deployment misconfiguration has prevented an authentication feature from being available in a group of customer environments, resulting in impact.

Next update by: Wednesday, May 18, 2022, at 5:30 PM UTC

Published Time: 17/05/2022 22:11:29

Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online.

Current status: We've completed redeploying the feature to all affected remaining environments. While we've received confirmation from multiple affected users that the impact is now remediated, our review of telemetry revealed errors continuing to persist for some environments. We're implementing additional tooling to extract logging that will assist us in determining why these errors persist and formulate appropriate steps to remediate these remaining environments.

Scope of impact: This issue may potentially affect any user attempting to utilize Graph APIs for SharePoint Online.

Root cause: A recent feature deployment misconfiguration has prevented an authentication feature from being available in a group of customer environments, resulting in impact.

Userlevel 7
Badge +20

Wow that is quite the issue.

Userlevel 7
Badge +20

If I get any further service notifications on this I’ll post back here. Microsoft haven’t provided any details on if this impacts any specific regions either, so the spread of impact is still unknown.

Userlevel 7
Badge +12

Thanks for posting this @MicoolPaul.

Userlevel 7
Badge +20

FINAL UPDATE

Published Time: 18/05/2022 17:15:00


Title: Users may see 'Access Denied' errors when using Graph APIs for SharePoint Online

User Impact: Users may have seen 'Access Denied' errors when using Graph APIs for SharePoint Online.

Final status: We've reverted the code regression and after monitoring the service and verifying with affected users, we've confirmed this action has remediated impact in full.

Scope of impact: This issue may have potentially affected any user attempting to utilize Graph APIs for SharePoint Online.

Start time: Sunday, May 15, 2022, at 11:00 PM UTC

End time: Wednesday, May 18, 2022, at 7:00 AM UTC

Root cause: A recent feature deployment misconfiguration had prevented an authentication feature from being available in a group of customer environments, resulting in impact.

Next steps:
- We're investigating why the recent feature deployment misconfiguration had prevented an authentication feature from being available in a group of customer environments to prevent this problem from happening again in the future.

This is the final update for the event.

Comment