Utilities and extensions useful for any Nuke builds originally separated from Nuke.Unreal.
Name comes from Nuka Cola of the Fallout franchise.
Build plugins for Nuke is a way to implicitly share build tasks which the main build tool can pick up automatically from anywhere within the project. For components which are independent of each-other but they don't individually make a complete project, like a composition of those are expressing the resulting piece of software, it might not make sense for them to have independent full fledged Nuke context for each of them. For these scenarios Nuke.Cola provides a way to discover C# scripts or C# projects following a specific convention anywhere inside the subfolders of the project (recursively). These plugins can be then just distributed and re-used along these components.
Practical justification / reasoning / motivation
This is originally developed for Unreal plugins where the main project's Nuke scripts or the pre-built independently distributed build-tools shouldn't explicitly know about the plugin composition of the project they need to work on in runtime. Unreal, or other non-dotnet project models might not have the capacity or cultural acceptance to rely on a Nuget infrastructure to distribute build scripts of these independent software components, which then could be referenced by the main project. Even if that would be plausable it would still take uncomfortable extra boilerplate for each of these software components. In case of a pre-built build tool based on a Nuke build script, this is the only way I know of to have dynamically composable software component specific build scripts considered.
Build plugins are discovered before the main entry point of Nuke, if the developer uses
public static int Main () => Plugins.Execute<Build>(Execute);instead of the traditional
public static int Main () => Execute<Build>(x => x.Compile);Note
- Your main Build class needs to be
publicfor this to work. - Currently it is not yet implemented to support an explicit default target, so when Nuke is executed without arguments, it will just print the help text.
The following kinds of plugins discovered this way:
*.nuke.csxstandalone C# script files.*.Nuke.csprojnamed C# projects.[ImplicitBuildInterface]tagged interfaces in the main build.
Scripts are better when there's a single file with few targets which don't need to interact with the main build class, projects are better for more elaborate scenarios and [ImplicitBuildInterface] can be used when <Compile Include="../**/*.nuke.cs" /> is specified for the build project.
In all cases build interfaces inheriting INukeBuild are picked up and their targets and parameters are added to the final Nuke build class. Read more about Nuke build interfaces (or "Build Components" as they call it). Targets of Plugins have complete freedom to interact with the entire build graph, especially when the build graph is expressed first in a Nuget package library (like Nuke.Unreal already gives Unreal plugins a lot to work with).
This is simply an interface defined in your main build project. It may not seem very useful until one factors in the following addition to your build csproj:
<ItemGroup>
<Compile Include="../**/*.nuke.cs" />
</ItemGroup>This means that without any further configuration one can put .nuke.cs files anywhere in their project and write scripts in the context of their placement. This is the easiest to configure method requiring the least bboilerplate but obviously it doesn't work on pre-built nuke build-tools.
except...
if one creates a csproj based build plugin which sole purpose is to include source files that way. In that case the prebuilt build tool can discover .nuke.cs files provided when compiling the csproj plugin.
Example:
using Nuke.Common;
using Nuke.Unreal;
using Nuke.Cola;
using Serilog;
[ImplicitBuildInterface]
public interface IExtraTargets : INukeBuild
{
Target TestPlugin => _ => _
.DependentFor<UnrealBuild>(b => b.Generate)
.Executes(() =>
{
Log.Information($"Hello from folder {this.ScriptFolder()}");
});
}Open for detailed explanation:
using Nuke.Common;
using Nuke.Unreal;
using Nuke.Cola;
using Serilog;
// This attribute is necessary so other optional build components wouldn't get used unexpectedly
[ImplicitBuildInterface]
// The build component interface should only declare members with default implementations
// as there's no manual way to provide those in the implementing intermediate build class.
public interface IExtraTargets : INukeBuild
{
// Define your targets or parameters freely and connect them with the build graph
// OR the developer can explicitly call them with `nuke test-plugin` in this case
Target TestPlugin => _ => _
// Automatically run this target before the Generate target is invoked.
// `UnrealBuild` is a base build class providing common Unreal related targets and
// parameters, including `Generate`.
.DependentFor<UnrealBuild>(b => b.Generate)
// Finally declare what this target should actually do when invoked
.Executes(() =>
{
// Use this.ScriptFolder() to work with this file's location
Log.Information($"Hello from folder {this.ScriptFolder()}");
});
}Example:
#r "nuget: md.Nuke.Unreal, 2.0.5"
using Nuke.Common;
using Nuke.Unreal;
using Serilog;
public interface IExtraTargets : INukeBuild
{
Target TestPlugin => _ => _
.DependentFor<UnrealBuild>(b => b.Generate)
.Executes(() =>
{
Log.Information($"Hello from folder {this.ScriptFolder()}");
});
}Open for detailed explanation:
#r "nuget: md.Nuke.Unreal, 2.0.5"
using Nuke.Common;
using Nuke.Unreal;
using Serilog;
// The build component interface should only declare members with default implementations
// as there's no manual way to provide those in the implementing intermediate build class.
public interface IExtraTargets : INukeBuild
{
// Define your targets or parameters freely and connect them with the build graph
// OR the developer can explicitly call them with `nuke test-plugin` in this case
Target TestPlugin => _ => _
// Automatically run this target before the Generate target is invoked.
// `UnrealBuild` is a base build class providing common Unreal related targets and
// parameters, including `Generate`.
.DependentFor<UnrealBuild>(b => b.Generate)
// Finally declare what this target should actually do when invoked
.Executes(() =>
{
// Use this.ScriptFolder() to work with this file's location
Log.Information($"Hello from folder {this.ScriptFolder()}");
});
}You can put a *.nuke.csx file anywhere and it will be picked up as a Build Plugin. Nuke.Cola will also configure VSCode for C# scripts auto-completion support as common courtasy. In order for VSCode to pick up nuget references use .NET: Restart Language Server via the command palette (or Omnisharp: Restart OmniSharp in case that fallback is used). Debugging plugins require you to modify .vscode/launch.json and run the desired targets/parameters as startup. Mutliple scripts with the same name in different folders can co-exist as long as they define unique interface names.
You can put a *.Nuke.csproj named project anywhere and the build script using Nuke.Cola will pick it up. Simplest way to do it is via dotnet command line:
> dotnet new classlib --name MyPlugin.Nuke
then add Nuke.Cola Nuget package (and your own project's specific shared build components):
> cd .\MyPlugin.Nuke
> dotnet package add md.Nuke.Cola
and then you can proceed as with any other dotnet class library.
Note
Unlike scripts, each C# project build plugin needs to be named uniquely in one project.
There are cases when one project needs to compose from one pre-existing rigid folder structure of one dependency to another rigid folder structure of the current project. For scenarios like this Nuke.Cola provides ImportFolder build class extension method which will copy/link the target folder and its contents according to some instructions expressed by either an export.yml file in the imported folder or provided explicitly to the ImportFolder extension method.
Practical justification / reasoning / motivation
In Unreal Engine a code plugin can only be uploaded to the Marketplace (or now Fab) if it doesn't depend on any other code plugins or anything else outside of the archive the seller provides Epic for distribution. This makes however sharing code between these plugins non-trivial.
Simply duplicating code among plugins is good enough until the user acquires two or more of these plugins using shared code with the same module names. Unreal modules names need to be unique in a global scope because Epic doesn't like namespaces for pragmatic reasons. To work around this while allowing duplicated code to be referenced in Unreal projects without ambiguity I cooked up an automatic solution which may need lot's of explanation but provides easy setup maintaining freedom for modularity.
Folder composition allows couple of more things:
- Maintaining the above mentioned shared code in a central location without that needing to know where they might be shared
- Linking to only couple of subfolders of a submoduled monorepo in your root project.
For example the following target:
[ImplicitBuildInterface]
public interface IImportTestFolders : INukeBuild
{
Target ImportTestFolders => _ => _
.Executes(() =>
{
var root = this.ScriptFolder();
var target = root / "Target";
var thirdparty = root / "ThirdParty";
this.ImportFolders("Test"
, (thirdparty / "Unassuming", target)
, (thirdparty / "FolderOnly_Origin", target)
, (thirdparty / "WithManifest" / "Both_Origin", target / "WithManifest")
, (thirdparty / "WithManifest" / "Copy_Origin", target / "WithManifest")
, (thirdparty / "WithManifest" / "Link_Origin", target / "WithManifest")
);
});
}will process/import the file/folder structure on the left to the file/folder structure on the right.
ThirdParty -> Target
├───FolderOnly_Origin -> ├───FolderOnly_Test <symlink>
│ SomeFile_B.txt │ *
├───Unassuming -> ├───Unassuming <symlink>
│ SomeFile_A.txt │ *
└───WithManifest -> └───WithManifest
├───Both_Origin -> ├───Both_Test
│ │ export.yml │ │ -
│ │ SomeModule_Origin.build.txt -> │ │ SomeModule_Test.build.txt
│ ├───ExcludedFolder │ │ -
│ │ SomeFile_Excluded.txt │ │ -
│ ├───Private -> │ ├───Private
│ │ │ ModuleFile_Origin.cpp.txt -> │ │ │ ModuleFile_Test.cpp.txt
│ │ └───SharedSubfolder -> │ │ └───SharedSubfolder <symlink>
│ │ SomeFile.cpp.txt │ │ *
│ └───Public -> │ └───Public
│ └───SharedSubfolder -> │ └───SharedSubfolder <symlink>
│ SomeFile.h.txt │ *
├───Copy_Origin -> ├───Copy_Test
│ │ export.yml │ │ -
│ ├───Foo -> │ ├───Foo
│ │ │ Foo.bar -> │ │ │ Foo.bar
│ │ └───Bar_Origin -> │ │ └───Bar_Test
│ │ A.txt -> │ │ A.txt
│ └───Wizzard -> │ └───Wizzard
│ │ B.txt -> │ │ B.txt
│ └───Ech -> │ └───Ech_Test
│ C_Origin.txt -> │ C_Test.txt
└───Link_Origin -> └───Link_Test
│ export.yml │ -
├───Foo -> ├───Foo
│ │ Foo.bar -> │ │ Foo.bar
│ └───Bar_Origin -> │ └───Bar_Test <symlink>
│ A.txt │ *
└───Wizzard -> └───Wizzard
│ B_Origin.txt -> │ B_Test.txt <symlink>
└───Ech_Origin -> └───Ech_Test
C_Origin.txt -> C_Test.txt <symlink>
Note
The resulting file/folder structure is also controlled by the export.yml files which content is not explicitly spelled out here, and is discussed further below.
To break it down:
A regular folder (like Unassuming) with no extra import instructions provided will be just symlinked.
Folders and files containing a preset suffix (by default Origin with either _, . and : as separators) can be replaced by a suffix chosen by the importing script (in this case Test). Files/Folders linked or copied will have this suffix replaced at their destination. (see FolderOnly_Origin -> FolderOnly_Test)
Folders can dictate the intricacies of how they're shared this way with a simple manifest file called export.yml. For example the one in Both_Test does
link:
- dir: Private/SharedSubfolder
- dir: Public/SharedSubfolder
- dir: Some/Deep/Folder/For/Some/Reason
as: MyNiceFolder
copy:
- file: "**/*_Origin.*"
procContent: trueLinking folders is straightforward and globbable. Target folders and parent folders (up until export root) is processed for suffixes at destination same as when copying folders.
When linking files, globbed files are individually symlinked with suffix processing on file/folder name.
Copying folders recursively have the same folder name processing but doesn't touch its contents (not even file/folder name processing)
When copying files (including when they're globbed) their content can be also processed for suffixes if procContent is set to true. Copying an entire folder recursively but with all the file/folder names processed can be done by simply doing recursive globbing -file: "MyFolder/**". The reasoning behind this design is suggesting performance implications, that each file is individually treated.
The destination relative path and name can also be overridden with as:. This works with globbing as well where the captures of * and ** can be referred to as $N where N is the 1 based index of the wildcards. For example
- file: Flatten/**/*.txt
as: Flatten/$2.txtWhere all the files inside the recursive structure of subfolder Flatten is copied/linked into a single-level subfolder. $2 in as: indicates it uses the second wildcard (the one at *.txt).
NOTE about string values in YAML containing *
* in YAML has special meaning. Therefore string values containing * needs to be single or double quoted. Therefore
copy:
- file: **/*_Origin.* # ERRORcopy:
- file: "**/*_Origin.*" # OK
- file: '**/*_Origin.*' # OKNuke.Cola comes with couple of useful extensions to Nuke's Tool delegate.
Whenever the Nuke Tooling API gives you a Tool delegate it is a clean slate, meaning you need to provide it your arguments, environment variables, how one reacts to its output etc. With the intended usage once these parameters are given to the Tool delegate it immediately executes the tool it represents.
However there are cases when multiple tasks with one tool requires a common set of arguments, environment variables or any other parameters Tool accepts. In such cases the API preferably would still provide a Tool delegate but the user of that API shouldn't need to repeat the boilerplate setup for that task involving the tool. The solution Nuke.Cola provides is the With extension method which allows to combine together the parameters Tool accepts but in multiple steps. See:
public static Tool MyTool => ToolResolver.GetPathTool("my-tool");
public static Tool MyToolMode => MyTool.With(arguments: "my-mode");
public static Tool WithMyEnvironment(this Tool tool) => tool.With(environmentVariables: SomeCommonEnvVarDictionary);
// ...
MyTool("args"); // use normally
MyToolMode("--arg value"); // yields `my-tool my-mode --arg value`
MyToolMode.WithMyEnvironment().WithSemanticLogging()("--arg value"); // excercise for the readerBuild steps which may require random set of tools can provide the means to set up those tools before usage for the system. Simply using:
ToolCola.Use("cmake");If cmake is not found in PATH, then Nuke.Cola will first attempt to install it via OS specific package managers. Finally it returns an ValueOrError wrapper further letting the developer to react to errors in a fluent way. Consider bundled tools
ToolCola.Use("pip", comesWith: () => ToolCola.Use("python").Get());which will first try to setup python (if it isn't already) and then attempt to get pip.
Note
the .Get() there will return the Tool without a wrapper or will throw all the previous errors accumulated if ToolCola.Use didn't manage to fetch the desired tool. In this context however if it fails, it will be caught and recorded into ValueOrError chain for pip and go onto the next attempt.
See ErrorHandling class for how ValueOrError is implemented.
Nuke.Cola comes with explicit support of some tools
- Python
- VCPKG
- XMake/XRepo
- See
XRepoItemfor parsed package information
- See
Build scripts can get complex enough that it is hard to fisrt grasp the options it can give to the user especially ones which dynamically import Build Plugins. Of course we have --help + --plan or parameters.json + profiles features Nuke provides, but a nice interactive UI can help much more with team adoption, especially one which shows relations of which parameters are being used by which Nuke Target.
So far this is only there to aid Nuke adoption and select targets and related parameters to run for people who're not familiar with Nuke yet. More visualizations to mirror target relations and support more Nuke goodies might come in a distant future™.