make it easier to change app to single threaded stages

[hymm]

Objective

• Users should have an easier way of switching their threading model.
• I also want this for fixing The test correct_transforms_when_no_children spuriously fails. #5285 to make it easy to switch the panicking test to single threaded.

Solution

• change add_default_stages to take an option to pick threading model of the default stages
• added constructors on App, App::single_threaded and App::multithreaded that allow users to construct an app with default stages as single threaded or multi threaded.

Migration Guide

add_default_stages now takes an option.

Before

App::empty().add_default_stages();

After

App::empty().add_default_stages(AppThreading::Multi);

Notes

• Not sure if App::new_with_threading_options should just replace App::new(). It would mean we'd need to change all the examples to use App::default() instead of new added separate constructors with the option set instead

[nicopap]

I think get_stage should be defined somewhere else). It could very well replace App::new, since, after all, it's in general used only once in a game, which I don't think is too burdensome to update. Though a deprecation warning period + a migration guide pointing out App::default would be necessary.

[nicopap]

Idea: What about adding a Default to AppThreading and calling AppThreading::default() here? Just a suggestion

[nicopap]

Also I had no idea add_default_stages was a thing. :D

[hymm]

I added constructors for App::single_threaded() and App::multi_threaded() and made App::new_with_threading_options private, instead of adding an option to new(). Not sure about the naming here, since you can still use query.par_for_each and the task pools to do multi threading.

[cart]

I think ideally this is an "app setting", which we use for all app stages (by default). Ex: replace instances of add_stage(Label, stage_reference) with add_default_stage(Label). And with stageless on the horizon, this would need to be a global app setting anyway (ex: your only options are to use a single threaded app executor or a multithreaded app executor).
What are your thoughts on waiting for stageless before defining this api vs building this api (with stageless in mind)?

[hymm]

I'm fine with closing this out until after stageless.

[hymm]

And with stageless on the horizon, this would need to be a global app setting anyway

This could be a per schedule setting in stageless.