Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Boost AAD performance by replacing Cmdlets with slow load times #1196

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

tkol2022
Copy link
Collaborator

@tkol2022 tkol2022 commented Jul 2, 2024

🗣 Description

*This PR replaces #1094 which became too difficult to deal with since there were heavy merge conflicts - reference that PR for additional details.

The gist of this pull request was to modify the AAD provider so that we replace the Cmdlets in the Microsoft.Graph.Beta.Identity.Governance Powershell module with direct calls to MS Graph REST APIs. We found that the offending Powershell module takes a significant amount of time to load into memory when you call ScubaGear the first time in a fresh Powershell window. The good news was that the RESP API returns the same fields as the Cmdlet, except that the field names begin with a lower case letter so that is why you see those respective changes in the Rego and unit/functional tests as well as references in the AAD provider.

closes #816

🧪 Testing

The original code changes were tested against the E5, G5, G3 and GCC High tenants. Additional testing should be done by the reviewers to ensure the code performs as expected and doesn't crash.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • PR targets the correct parent branch (e.g., main or release-name) for merge.
  • Changes are limited to a single goal - eschew scope creep!
  • Changes are sized such that they do not touch excessive number of files.
  • All future TODOs are captured in issues, which are referenced in code comments.
  • These code changes follow the ScubaGear content style guide.
  • Related issues these changes resolve are linked preferably via closing keywords.
  • All relevant type-of-change labels added.
  • All relevant project fields are set.
  • All relevant repo and/or project documentation updated to reflect these changes.
  • Unit tests added/updated to cover PowerShell and Rego changes.
  • Functional tests added/updated to cover PowerShell and Rego changes.
  • All relevant functional tests passed.
  • All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

  • PR passed smoke test check.

  • Feature branch has been rebased against changes from parent branch, as needed

    Use Rebase branch button below or use this reference to rebase from the command line.

  • Resolved all merge conflicts on branch

  • Notified merge coordinator that PR is ready for merge via comment mention

✅ Post-merge checklist

  • Feature branch deleted after merge to clean up repository.
  • Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

@tkol2022 tkol2022 self-assigned this Jul 2, 2024
@tkol2022
Copy link
Collaborator Author

tkol2022 commented Jul 2, 2024

For reviewers: How to test the performance improvement

The performance improvement when running AAD is mostly noticeable when the tool executes for the first time in a newly spawned Powershell window. This is because the slowness that we observed in the Microsoft.Graph.Beta.Identity.Governance dll was during its load into memory, which occurs only the first time you call one of its Cmdlets in a new Powershell window.
The steps below describe how you can test to ensure that you also see an improvement.

Use this code to take a time measurement (tailor the parameter values to your environment):

Import-Module -Name .\PowerShell\ScubaGear
$ExecutionTime = Measure-Command { Invoke-Scuba -ProductNames aad -CertificateThumbPrint "33dd41a761f434f12ee8ac3ff771347597435585" -AppID "ac691fde-1ff0-493f-be3d-03d9ba6cb891" -Organization "TEST_TENANT.onmicrosoft.com" }
$ExecutionTime
  1. Download a copy of the main branch and run the measurement code above 5-10 times. Remember to take each measurement in a new Powershell window. Compute an average time.
  2. Download a copy of the branch associated with this PR and repeat step 1.
  3. Log the average time of the main branch and this branch in this PR.

@tkol2022
Copy link
Collaborator Author

tkol2022 commented Jul 2, 2024

@julianjburgos Test against the E5 tenant
@dagarwal-mitre Test against the G5 and G3 tenants

Follow the instructions in the prior comment above.

@dagarwal-mitre
Copy link
Collaborator

dagarwal-mitre commented Jul 3, 2024

@julianjburgos Test against the E5 tenant @dagarwal-mitre Test against the G5 and G3 tenants

Follow the instructions in the prior comment above.

Averages:

Main New Branch
G3 36.7 Seconds 38.3 Seconds
G5 98.1 Seconds 54.4 Seconds

Copy link
Collaborator

@dagarwal-mitre dagarwal-mitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment about average times, the time for the G3 tenant on average was slightly slower in testing

@tkol2022
Copy link
Collaborator Author

tkol2022 commented Jul 8, 2024

See comment about average times, the time for the G3 tenant on average was slightly slower in testing

The Powershell code that was optimized affects tenants that are G5 and above.

@julianjburgos
Copy link
Collaborator

@julianjburgos Test against the E5 tenant @dagarwal-mitre Test against the G5 and G3 tenants

Follow the instructions in the prior comment above.

Averages:

Main New Branch
E5 127.1 Seconds 99.2 Seconds

Copy link
Collaborator

@julianjburgos julianjburgos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look at comment above with average times for E5

@tkol2022
Copy link
Collaborator Author

tkol2022 commented Jul 9, 2024

@nanda-katikaneni ready for merge testing.

@nanda-katikaneni nanda-katikaneni merged commit aa7733b into main Jul 10, 2024
27 of 33 checks passed
@nanda-katikaneni nanda-katikaneni deleted the 816-aad-performance-fix-revamped branch July 10, 2024 13:40
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prototype a performance redesign of the AAD provider
4 participants