[BUG] - Memory consumption creating folder with Resolve-PnPFolder

[verdeterrejim]

Notice

Many bugs reported are actually related to the PnP Framework which is used behind the scenes. Consider carefully where to report an issue:

1. Are you using Invoke-PnPSiteTemplate or Get-PnPSiteTemplate? The issue is most likely related to the Provisioning Engine. The Provisioning engine is not located in the PowerShell repo. Please report the issue here: https://github.com/pnp/pnpframework/issues.
2. Is the issue related to the cmdlet itself, its parameters, the syntax, or do you suspect it is the code of the cmdlet that is causing the issue? Then please continue reporting the issue in this repo.
3. If you think that the functionality might be related to the underlying libraries that the cmdlet is calling (We realize that might be difficult to determine), please first double check the code of the cmdlet, which can be found here: https://github.com/pnp/powershell/tree/master/src/Commands. If related to the cmdlet, continue reporting the issue here, otherwise report the issue at https://github.com/pnp/pnpframework/issues

Reporting an Issue or Missing Feature

Issue with memory consumption creating folder with Resolve-PnPFolder

Expected behavior

When running a script to process thousands of list items in one library, folders are being created in a new library to prepare for
moving the documents to the new library within the newly created folder. As the script is running I expect that memory
consumption will stay stable and not continually increase.

Actual behavior

As the script is running, memory continues to increase until the machine runs out of memory.

Steps to reproduce behavior

This is the simple script that will cause memory to continually increase:

Get-PnPListItem -Connection $SPConnection -List $LibraryTitle  `
  -PageSize 500 -ScriptBlock { Param($items) $items.Context.ExecuteQuery() } | `
  ForEach-Object {  
    $targetFolderSiteRelUrl = # site relative reference to folder path in different library (where folder doesn't exist)
    Resolve-PnPFolder -Connection $SPConnection -SiteRelativePath $targetFolderSiteRelUrl
  }

What is the version of the Cmdlet module you are running?

1.9.0

Which operating system/environment are you running PnP PowerShell on?

• Windows

[jackpoz]

Resolve-PnPFolder returns the folder, so your powershell instance is keeping an object with the created folder to show as output at the end.
I would suggest trying with Resolve-PnPFolder | Out-Null . See https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/out-null?view=powershell-7.2 for documentation about Out-Null.

This sounds more like a powershell issue than a PnP PowerShell issue.

[verdeterrejim]

I tried piping to Out-Null but still experienced the memory consumption issue. I did a memory profile and it's showing that ObjectPathIdentity objects are being retained in the ContextCache. It's the same issue as reported here: pnp/PnP-PowerShell#1182

.

[jackpoz]

On slightly unrelated note, I ran a script that tried to connect to 210'000 sites one at a time, load $web.LastItemUserModifiedDate, output it to a csv (with -Append) and disconnect from the current site. After 70'000 sites the ram usage reached 3.5 GB and the script got stuck due to memory pressure.

I will try getting a memory dump on the next run to check if it's caused by the same ObjectPathIdentity. Not sure what would be a good approach to fix this, automatically free'ing the cache based on some rules or allowing to manually clear it or both or yet another way.

[gautamdsheth]

@jackpoz , @verdeterrejim - can you guys try the script with tomorrow's nightly build ? Have managed to reduce usage of ExecuteQuery , so should definitely help and speed up folder creation.

[gautamdsheth]

@jackpoz / @verdeterrejim - any feedback on this ? Would love to here your thoughts on this.

[verdeterrejim]

Hi Gautam. I will try to test using Resolve-PnPFolder in a week or two. However, the script has a workaround and is now positioned for a production run so I won’t be able to change it now. I’ll try to do a test after the production run. From: Gautam Sheth ***@***.***> Sent: June 17, 2022 12:25 AM To: pnp/powershell ***@***.***> Cc: Steve Hart ***@***.***>; Mention ***@***.***> Subject: Re: [pnp/powershell] [BUG] - Memory consumption creating folder with `Resolve-PnPFolder` (Issue #1637) @jackpoz<https://github.com/jackpoz> / @verdeterrejim<https://github.com/verdeterrejim> - any feedback on this ? Would love to here your thoughts on this. — Reply to this email directly, view it on GitHub<#1637 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALYIUUPCJ6H5GMJTMND2RNTVPQR5XANCNFSM5OPETJRQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[jackpoz]

@gautamdsheth please note my case was about loading $web.LastItemUserModifiedDate for few thousands of sites and not creating any folder. I have not had time yet to inspect the .NET memory dump.

[jackpoz]

I have not had time to inspect the .NET memory dump but I ran the inventory script again (just reading data from 220'000 site collections) and this time the RAM usage stayed low. I don't use Resolve-PnPFolder but some other unrelated changes might have fixed my issue.