Hands-on feasibility analysis calling M365 REST APIs directly without Powershell dependency modules

[tkol2022]

💡 Summary

This is a feasibility analysis via hands-on prototyping.

Ultimately ScubaGear is limited to whatever configuration fields the underlying Powershell modules offer. In some cases we have found that the fields we need are available in a back-end REST API even though we cannot get them via modules such as Sharepoint.Powershell. Therefore we investigated to see:

• How could ScubaGear get M365 to mint authentication tokens that we could use when making REST API calls to the back-end?
• Can we code a solution for both interactive authentication and non-interactive authentication (using a client certificate)?
• What is the feasibility of incorporating such a solution in ScubaGear including long-term maintenance impacts?

This work is an extension of previous work that was performed to call Sharepoint Online REST APIs directly without going through the PnP.Powershell or Sharepoint.Powershell modules that ScubaGear currently relies on. ScubaGear currently cannot get all of the configuration fields necessary to perform a full audit of the M365 tenant either because the underlying module does not include the fields (Sharepoint.Powershell) or we cannot use the latest version of the module (PnP.Powershell) because it requires Powershell 7.

Some of the previous related issues are:
#958 which proved that calling the Sharepoint REST API directly can acquire numerous fields that are currently missing from ScubaGear.
#957 which provides a proof of concept code branch that demonstrates how to call .NET MSAL authentication APIs to perform interactive user authentication and get a token that can be used to directly call Sharepoint REST APIs.

Lack of cmdlet support regarding interactive authentication to call Sharepoint REST APIs: The PnP.Powershell module offers an Invoke-PnPSPRestMethod cmdlet which can be used to make REST API calls directly to Sharepoint sites with a certificate without the need for custom MSAL authentication logic, but there is no equivalent in the Sharepoint.Powershell module for interactive authentication. We cannot use PnP.Powershell with interactive authentication since it requires either an existing PnP multi-tenant app that is overpermissioned or the user must create a custom app in their tenant which is not currently required for ScubaGear users with interactive authentication. We examined cmdlets outside of PnP.Powershell for interactive authentication such as Connect-MgGraph, but there does not seem to be a way to call Connect-MgGraph with Sharepoint as the target application for the token and subsequently call a Sharepoint REST API.

Precedence for direct REST API calls: There is precedence in ScubaGear for calling REST APIs directly. In Entra Id, earlier this year, ScubaGear was modified to directly call REST APIs instead of going through the MS Graph cmdlets due to performance problems with the cmdlets. We made these modifications for a subset of the Graph cmdlets that the Entra Id baseline was using. For Entra Id, we didn't need to worry about writing custom code to acquire authentication tokens and pass them to a Powershell function that makes REST API calls because that was handled by Connect-MgGraph (which handles the magic of getting the tokens) and Invoke-MgGraphRequest (which calls the REST API and uses the underlying token).

Note: If the Scuba team decided to implement a solution to directly call Sharepoint APIs then ScubaGear would no longer need the PnP.Powershell and Sharepoint.Powershell dependencies and there would be a single code path in the Sharepoint export provider.

Implementation notes

• Develop code that calls the .NET MSAL assembly to perform non-interactive authentication via certificate and then use the acquired token to call the back-end Sharepoint REST API
• Develop code that calls the .NET MSAL assembly to perform interactive user authentication with MFA and then use the acquired token to call the back-end Sharepoint REST API
• Schedule a demo to the development team and answer questions
• Post the results of the feasibility study
• Create any necessary follow-up action issues if they are needed
• Give the team a copy of the prototype source code and dependency files
• Make a backup of the code in an external repository

[tkol2022]

Thank you

I wanted to thank the authors of the following blog post and GitHub repo for inspiration and code reference which was invaluable for the development of the source code associated with this issue.

• https://blog.icewolf.ch/archive/2023/10/16/testing-psmsalnet-because-msal-ps-has-been-archived/
• https://github.com/SCOMnewbie/PSMSALNet
• https://github.com/SCOMnewbie/PSMSALNet/blob/main/source/Public/Get-EntraToken.ps1
• @twneale from issue Proof of Concept - Sharepoint provider MSAL authentication and REST API to get missing fields and uniform API invocation #957

[tkol2022]

Prototype for directly calling Sharepoint REST APIs with the Microsoft MSAL library (certificate authentication)

This prototype relies on the Microsoft Authentication Library (MSAL) which is Microsoft's modern library that enables application developers to acquire tokens in order to call web APIs. Specifically the Powershell code relies on the Microsoft.Identity.Client Namespace for .NET. The assemblies are Microsoft.Identity.Client and Microsoft.IdentityModel.Abstractions.

Dependencies

I took care of the installation by downloading recent versions of Microsoft.Identity.Client.dll and Microsoft.IdentityModel.Abstractions.dll and embedding them in the source code folder. I got them from the official Nuget repository. If ScubaGear were to use these assemblies we would need to write code in our setup routines to download the package and then load the version of the dll files that match the .NET version on the system ScubaGear is running on.

SharepointAPIWithCertificate.ps1 (demo script)

This script calls the .NET MSAL AcquireTokenForClient method to authenticate and acquire a token using client certificate authentication. You need to have an Entra Id registered application like the Scuba Functional Test Orchestrator created in the tenant and your client certificate configured as a credential in the application. Your client certificate must be in the cert:\CurrentUser\My section in the Windows certificate registry on the system you are running on. Once the authentication token is acquired, the script passes the token to an API call against a Sharepoint REST method. The script then outputs a JSON document to the console that contains the Sharepoint tenant configuration settings received from the REST API.

Permissions

The registered Entra Id application that you use must have the API permission shown in the screenshot below to be able to get data from Sharepoint Online.

Example usage:

This example assumes you have a M365 tenant with the domain your365tenant.onmicrosoft.com

.\SharepointAPIWithCertificate.ps1 -ClientId ac391fde-1ff0-222f-be3d-03d9ba6cb987 -CertificateThumbprint 93fd41a761f434f12ee8ac3ff771347597435696 -TenantDomainPrefix your365tenant

SampleSharepointTenantConfig.json

This is a sample Sharepoint tenant configuration settings output file generated by the script above.

Snippet of example output JSON below.

[tkol2022]

Study Results

Overall my experience with the MSAL libraries via Powershell was positive. Loading the respective MSAL dll dependencies and minting a token via a call to the authentication endpoint is about 10 lines of code. Getting that 10 lines working right was tricky and took a little time.

Once I got it working I was able to call a Sharepoint REST endpoint to retrieve the tenant settings and identify that the endpoint provides fields that are missing from the Microsoft.Online.SharePoint.PowerShell cmdlets. I was also able to mint tokens and call MS Graph endpoints using the same foundational authentication code, which proves that directly accessing the MSAL library may help Scuba authenticate to disparate services across M365 with a uniform code base (today ScubaGear uses a hodge podge of different authentication cmdlets from the Powershell modules).

Benefits

• Ability to see the underlying response fields from the M365 endpoints (and therefore fully discover what they have to offer beyond what is exposed by the Powershell cmdlets)
• Efficiency gained by bypassing middleware cmdlets (some cmdlets have proved to be resource hogs in the past)
• Get a good feel for what it takes to directly call core Microsoft authentication libraries for M365 endpoints from Powershell
• The Sharepoint endpoint returns JSON data so if we were to call it directly from ScubaGear we would not have to change much in the Powershell and Rego code that references specific fields. In most cases it is only the case of the first letter in a field name that would change whether we use cmdlets (which we are doing now) versus call directly from MSAL (the prototype). The structure of the JSON appears similar.

Challenges

The toughest part with integrating this code is going to be managing the .NET dependency packages. ScubaGear would need to download a specific (known working version) of the .NET MSAL packages from the NuGet repo and then reference the files from ScubaGear. Since the NuGet packages each contain different dll files based on the .NET version you are running, our code would need to detect what system it was running on (and the Powershell version) in order to dynamically load the correct dll. Although this is not an overly challenging requirement, it can make the ScubaGear dependency management a little more complex versus what we do today which is simply installing Powershell modules (no dynamic linking needed in our code).

Limitations

The limitations below were due to a decision that I made on how much time to spend on the prototype to get a benefit out of it.

• I did not prototype authenticating to Teams and Exchange Online
• I did not prototype seeing if I could reduce the number of sign-in popup windows when ScubaGear users authenticate interactively to multiple M365 services during the same run (reducing these prompts may be possible with MSAL code by minting tokens silently although I am not sure without prototyping it)

[tkol2022]

Code Snippet

This code sample was taken from the script that authenticates via a service principal and then leverages the token to call a Sharepoint REST API. The most important lines are the following:

• The line that calls AcquireTokenForClient to mint a token for Sharepoint using a client certificate with a service principal
• The line that sets up the HTTP authentication header with a bearer token using the variable $sharepointAuthToken.AccessToken

$currentDir = $PSScriptRoot
try {
    $assembly = [Reflection.Assembly]::LoadFrom("$currentDir\Microsoft.IdentityModel.Abstractions.dll")
    # $assembly.FullName
    $assembly = [Reflection.Assembly]::LoadFrom("$currentDir\Microsoft.Identity.Client.dll")
    # $assembly.FullName
} catch {
    Write-Warning "Make sure the files Microsoft.IdentityModel.Abstractions.dll and Microsoft.Identity.Client.dll are in the current folder"
    throw
    return
}

# Setup some necessary variables
$Tenant = "$TenantDomainPrefix.onmicrosoft.com"
$Authority = "https://#.microsoftonline.com/$Tenant"

# The ConfidentialClientApplicationBuilder is simply a helper class that sets up some parameters
$ClientApplicationBuilder = [Microsoft.Identity.Client.ConfidentialClientApplicationBuilder]::Create($ClientId)
$ClientCertificate = Get-Item "cert:\CurrentUser\My\$CertificateThumbprint"
$ClientApplicationBuilder = $ClientApplicationBuilder.WithCertificate($ClientCertificate)
$ClientApplicationBuilder = $ClientApplicationBuilder.WithAuthority($Authority)
# Build an instance of the IConfidentialClientApplication interface which is then used to get the token 
$ConfidentialClientApp = $ClientApplicationBuilder.Build()


#####################################################################
##### This first section provides an example for calling a Sharepoint REST API.
##### It will output a JSON document containing the Sharepoint tenant configuration settings.
#####
# Acquire a token for Sharepoint
$SharepointAdminSite = "https://$TenantDomainPrefix-admin.sharepoint.com"
[string[]] $SharepointScopes = @("$SharepointAdminSite/.default")

try {
    $sharepointAuthToken = $ConfidentialClientApp.AcquireTokenForClient($SharepointScopes).ExecuteAsync().GetAwaiter().GetResult()
    # Write-Host "Sharepoint Access Token Acquired: $($sharepointAuthToken.AccessToken)"
} catch {
    Write-Host "Failed to acquire token: $($_.Exception.Message)"
    return
}

$SharepointConfigEndpoint = "$SharepointAdminSite/_vti_bin/client.svc/ProcessQuery"
$SharepointBody = @'
  <Request AddExpandoFieldTypeSuffix="true" SchemaVersion="15.0.0.0" LibraryVersion="16.0.0.0" ApplicationName=".NET Library" xmlns="http://schemas.microsoft.com/sharepoint/clientquery/2009">
    <Actions>
      <ObjectPath Id="2" ObjectPathId="1" />
      <Query Id="3" ObjectPathId="1">
        <Query SelectAllProperties="true">
          <Properties>
            <Property Name="HideDefaultThemes" ScalarProperty="true" />
          </Properties>
        </Query>
      </Query>
    </Actions>
    <ObjectPaths>
      <Constructor Id="1" TypeId="{268004ae-ef6b-4e9b-8425-127220d84719}" />
    </ObjectPaths>
  </Request>
'@

# Call a Sharepoint API to fetch configuration data and pass the acquired token in the header
$headers =  @{
    "Authorization" = "Bearer $($sharepointAuthToken.AccessToken)"
    "Accept-Encoding" = "gzip, deflate"
    "Content-Type" = "text/xml"
    "User-Agent" = "ScubaGear"
}

try {
    $response = Invoke-RestMethod -Uri $SharepointConfigEndpoint -Headers $headers -Method Post -Body $SharepointBody
    Write-Host "Sharepoint API Call Successful. Response:"
    $response | ConvertTo-Json
} catch {
    Write-Host "Sharepoint API call failed: $($_.Exception.Message)"
}