RFC: Pre and Post Run API

[FrozenPandaz]

RFC: Pre and Post Tasks Execution API

Problem

Workspaces have no clear way of defining hooks which happen before and after tasks are executed.

Custom task runners have traditionally handled this functionality, but their APIs had unclear extension points and it created a lot of problems for us while evolving Nx. This new API provides a well-defined way to handle common task runner use cases without compromises.

Prior Art

This is akin to how npm has preinstall and postinstall hooks.

Example Use Cases

• Performing validation before executing tasks
• Generating files before executing tasks
• Starting external processes before executing tasks
• Mutating environment variables before running tasks
• Reporting task results to an API
• Doing things which were originally done before and after tasks are run in a custom tasks runner function

Solution

Nx Plugins will be able to define functions which will be executed prior and after Nx runs tasks. The API will look something like the following (open to feedback):

plugin.ts

import type { PreTasksExecutionContext, PostTasksExecutionContext } from '@nx/devkit';

interface PluginOptions {
  field: any;
}

export async function preTasksExecution(options: PluginOptions, context: PreTasksExecutionContext) {
  // Happens before tasks run
  console.log('prerun')
}

export async function postTasksExecution(options: PluginOptions, context: PostTasksExecutionContext) {
  // Happens after tasks run
  console.log('postrun', context.taskResults)
}

Like npm's preinstall and postinstall hooks, there are many use cases where this API could be useful. However, just as npm's preinstall and postinstall hooks do little to influence the actual installation of dependencies, these hooks will also have little to no influence on the actual execution of tasks. The preRun hook may be used to prevent Nx from executing tasks if some criteria is not met by throwing an error.

Extra remarks

The reason the API is not affecting our ability to evolve Nx and is much easier for us to support is that it doesn't augment how Nx actually executes tasks: scheduling, orchestration, distribution, caching, handling io, terminal output will all happen in between the pre and post hooks. The only way to directly affect the execution is via env variables. Everything else is done indirectly (creating files, starting processes etc)

Demo

I've opened a Draft PR with an implementation of this API: #29636. A version of Nx will be published for those interested in trying out the API before it is released.

Edits

I changed the name of the API to preTasksExecution and postTasksExecution I'll update the PR in a bit.

[Jordan-Hall]

I know you have your power packs. But I think a lot of custom runners was used for caching even by custom API. How can this be done with this

[FrozenPandaz]

This API does not augment actual execution of tasks including augment caching behavior. We have both Nx Cloud and Powerpack caches available to augment the behavior of caching. Nx Cloud is free with limits and Powerpack is also free for small organizations.

This API offers a different set of capabilities as detailed above. Discussion about caching can be done elsewhere.

[med8bra]

It seems nx team is fully committed on making remote cache their a subscription service, and to host your cache you will need to go through a paywall. Not sure what was the basis of this product decision, but I believe nx has more potential than just to be a payme for cache product.

Turborepo documents their Remote Caching features, documented the API specification, and allowed the community to implement self hosted cache solutions.

[ianldgs]

Couldn't the preRun of a task be used to fetch its cache from a remote location?

[Tommy228]

@ianldgs my understanding is that it’s possible to fetch data from a remote cache (assuming you get the project hash somehow) but they do not make it possible to stop the execution of a task in that situation. The proposed implementation only allows stopping the execution by throwing an error. That would mark the task as failed on a remote cache hit.

[ianldgs]

@Tommy228 why would you want to stop the execution of the task?
In my mind, you'd fetch the cache and let the task proceed as usual. Since there will be a cache hit, the task won't actually execute, but get the outputs (build artifacts and stdout) from the cache.
That way, other tasks (such as publishing), can use the outputs.

[joshlartz]

The pre and post hooks part makes sense. The only thing I'm still a bit unsure on is this comment

The only way to directly affect the execution is via env variables. Everything else is done indirectly (creating files, starting processes etc)

We have definitely used a custom task runner to influence how the default task runner is about to run in fairly trivial ways. It's very possible env variables will do the job, but I don't really know what all is going to be exposed that way in v21 to compare.

[FrozenPandaz]

Good question, let me explain in more detail. It's a minor adjustment you'd have to do when you migrate from the customTasksRunner API.

Previously, with the customTasksRunner API it was possible to do something like the following:

export function customTasksRunner(tasks, options, context) {
  return defaultTasksRunner(tasks, {
    ...options,
    skipNxCache: true
  }, context);
}

Because the preTasksExecution hook is more isolated from the tasks runner, an alternative way to still influence Nx is via the use of environment variables:

export async function preTasksExecution() {
  process.env.SKIP_NX_CACHE = 'true';
}

This will yield the same results.

I hope that explains it. If you have any issues implementing it, we can hop on another call to help out.

P.S. The isolation between the before and post tasks execution hooks and the task execution itself is what makes this API better than the customTasksRunner API from a maintainability standpoint because it means that the inputs for the tasks execution will no longer be public API. This separation will allow us to evolve the inputs of the tasks execution more freely in the future.

[HaasStefan]

Awesome work. Thank you so much for this!

1.)
Could you share some more details about the context type? I would need some additional data, such a:

• task run duration
• start, end time (performance.now() + performance.timeOrigin)
• status of the task
• cacheableOperations
• skipCache flag
• daemon flag
• watch flag
• verbose flag
• task arguments

2.)
It would be great if the privileges of the process in the hooks are the same as for the task itself. So, if I run with sudo nx build someApp, I want to be able to determine that in the hooks.

3.)
Something else, that could be useful is o collect some custom data in the pre-run hook and then pass that into the post-run hook

This is because I would like to be able to collect data, such as freemem (and more), when the task starts and use that in the post-run hook

[FrozenPandaz]

Thanks Stefan!

This is a preview of the planned API to start.
https://nx-dev-git-lifecycles-nrwl.vercel.app/nx-api/devkit/documents/PreTasksExecution
https://nx-dev-git-lifecycles-nrwl.vercel.app/nx-api/devkit/documents/PostTasksExecution

A majority of the information you're looking for is available on each TaskResult and Task:
https://nx-dev-git-lifecycles-nrwl.vercel.app/nx-api/devkit/documents/TaskResult#task
https://nx-dev-git-lifecycles-nrwl.vercel.app/nx-api/devkit/documents/Task

This includes:

• Start time
• End time
• task run duration (End time - Start time)
• Status
• Task Arguments (overrides)
  • watch flag is also an override

Other information you're looking aren't currently planned to be explicitly in the API such as:

• cacheableOperations
• daemon flag
• skipCache flag
• verbose flag

However, it doesn't have to be explicitly tied to this API for it to be available to be used in the API. For instance, we could perhaps add @nx/devkit util methods which will allow any script, generator, executor, or plugin to all easily get the above values. Imagine being able to use import { isVerbose } from '@nx/devkit'; But before lumping that de#to this one, I think it's best for that to be a separate discussion from this one. If you have thoughts about those, we can jump on another call with you.

Something else, that could be useful is o collect some custom data in the pre-run hook and then pass that into the post-run hook

This is because I would like to be able to collect data, such as freemem (and more), when the task starts and use that in the post-run hook

The pre execution hook will be run in the same process as the post execution hook. So to pass data from the pre hook to the post hook, you can either pass it via the global object or the module scope.

It would be great if the privileges of the process in the hooks are the same as for the task itself. So, if I run with sudo nx build someApp, I want to be able to determine that in the hooks.

Sorry I'll get back to you on this one.

[FrozenPandaz]

It would be great if the privileges of the process in the hooks are the same as for the task itself. So, if I run with sudo nx build someApp, I want to be able to determine that in the hooks.

Confirmed, the process the hooks are executed on are spawned under the same user the main Nx process is spawned under. So userInfo will reflect whether the user running the command is root or a particular user on the machine. 👍

[wSedlacek]

One big missing piece from the proposed API in the PR is in the preTasksExecution it doesn't provide any details about the task that is going to be run. It only includes the workspace configuration.

This makes the first step of the custom cache flow impossible.

1. Hash Inputs
2. Reterive
3. Run
4. Hash and store output

If the task details could simple be provided as part of the PreTasksExecutionContext then I think most people would be happy with this as a solution.

[ianldgs]

Agreed; it wasn't clear to me when I first read the proposal that it wouldn't have ANY information about which tasks are going to run.

There's almost no difference between this proposal and:

node pre-tasks.js;
nx run-many --targets=lint,test,build;
node post-tasks.js;

Except for:

• a little bit more information (result of the tasks); but this could just be a different output-style, like json, of the run-many that is piped to a script
• a plugin can define it; but a package could instruct users to run the pre/post scripts like the shell script above via npx 🤷

[FrozenPandaz]

The preTasksExecution hook is invoked at a point when Nx does not have an accurate Task Graph with complete Task information yet.

The previous lifecycle invoked beforeTasks right before the task is run. This would be too granular for the behaviors that this API allows for such as performing validation before tasks run.

Tasks information at this point before any tasks execute is not entirely complete. Tasks information is only truly complete the moment before the task itself runs. Tasks later in the task graph can have their inputs hashes changed by tasks earlier in the task graph.

This is the reason why there is no tasks information provided for these lifecycle hooks.

[FrozenPandaz]

Thank you all for your feedback on the API. We have multiple reports of this API being helpful and sufficient for them. I'll move forward getting this landed for you all soon!

[zargham-leanix]

@FrozenPandaz Will post task runner allow to mark the task as failed, let's say if status is not 'remote-cache' I want to fail the task itself.

[FrozenPandaz]

Yes, an error may be thrown in the postTasksExecution hook if the task results or something else is not as expected.

[Bielik20]

This looks great, this is exact thing I was hopping for almost 4 years ago: #4927 😄