Add project template build tests + CG reporting

[MackinnonBuck]

Adds project template create/restore/build tests and CG reporting.

Summary of changes:

• Added infrastructure for invoking .NET commands from test code
• Added a TemplateSandbox folder that gets used to build (and in the future, run) tests in an environment isolated from the rest of the repo
• Added a script that configures the local PowerShell environment to match that of template execution tests
  • This makes debugging generated tests locally a lot easier than it would otherwise have been
• Added tests that create, restore, and build variations of the AI Chat Web project template
  • Note: Test output does not get cleared until the next test run. This enables easy debugging of generated templates and allows template dependencies to get discovered for CG.
• Added a README explaining how to debug template execution tests
• Made the Microsoft.Extensions.AI.Templates.Test project an "integration test" so it runs separately from existing unit tests
  • This is required because template execution tests can only be run after the repo gets packed
• Updated pipelines to include a "Run integration tests" step that runs after packing the repo

Fixes #6322

Microsoft Reviewers: Open in CodeFlow

[RussKie]

🚀

[SteveSandersonMS]

Cool!

This is all pretty sophisticated. Did you take much of the infrastructure code from our other repos or is this all custom for dotnet/extensions? If it's custom, do you think we should look into ways to share it with dotnet/aspnetcore etc or is this unique to the scenarios in this repo?

Presumably this could be used as the basis for Playwright tests in the future, though that would also force us to provide working AI service endpoints in CI which is a whole other challenge. Right?

[RussKie]

/cc: @radical

[MackinnonBuck]

Did you take much of the infrastructure code from our other repos or is this all custom for dotnet/extensions?

It's mostly custom, but I did look into the approaches taken in the aspnetcore, sdk, and aspire repos to see if there was anything that could be borrowed. Thankfully, it's quite a lot simpler to get these kinds of tests working in the extensions repo because we don't need to mutate the local .NET SDK installation to execute template projects (unlike e.g., aspnetcore, which needs to make a copy of the SDK with the just-built shared framework swapped in, or aspire, which has to install a workload). In the extensions case, it's enough to specify an alternate NuGet package cache location and a custom template install directory. Therefore, it didn't seem worth it to do a direct copy/paste of existing solutions when the complexity isn't warranted.

The main candidate for shared logic seems to be the ability to reliably run the .NET CLI from tests. The sdk repo has the most sophisticated solution for this out of the repos I looked at (Microsoft.DotNet.Cli.Utils + Microsoft.NET.TestFramework), and I effectively used a very simplified version of that (similar to aspire, but simpler still). And aspnetcore sort of does its own separate thing.

That said, it's likely possible to create a shared utility that provides the most common functionality required by each repo (maybe something like what the SDK already has). I'd be curious to see what others think about this.

Presumably this could be used as the basis for Playwright tests in the future, though that would also force us to provide working AI service endpoints in CI which is a whole other challenge. Right?

Yes, right.

[radical]

This is what we have - https://github.com/dotnet/aspire/blob/main/tests/Aspire.Templates.Tests/NewUpAndBuildStandaloneTemplateTests.cs

It doesn't depend on a workload now, so doesn't need to modify local dotnet installs. It supports testing against multiple sdks.

[radical]

Playwright tests - https://github.com/dotnet/aspire/blob/main/tests/Aspire.Templates.Tests/TemplateTestsBase.cs#L173

[jeffhandley]

With this now in place, what are your thoughts on the existing snapshot tests? Changes to the snapshot tests should be a separate PR if you suggest any though.

[MackinnonBuck]

With this now in place, what are your thoughts on the existing snapshot tests? Changes to the snapshot tests should be a separate PR if you suggest any though.

I think snapshot tests are still valuable for catching unintended changes in template content, which these new tests might not do. Aside from adding more snapshot tests over time as we see fit, I don't have any suggestions for how they need to change yet 🙂

[jeffhandley]

This is great, @MackinnonBuck. The only comment I have left that I'd like you to address is that I don't know what specific side-effect of generating and building the template combinations gives component governance visibility into the dependencies.

Can you add a comment in the place or places where there are any subtle details that make that wire up correctly? Is it how the projects are built, the fact that they simply perform a restore, the output location, and/or other aspects that result in CG visibility?

[jeffhandley]

Excellent comments and clarity; thanks!