Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

RFC: npm workspaces - Working with workspaces #117

Closed
wants to merge 9 commits into from
188 changes: 188 additions & 0 deletions accepted/0000-workspaces-part2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# npm workspaces: Running commands

## Summary

Introduces basic support to running **npm commands** across nested **workspaces**.

## Motivation

The need for running **npm commands** across nested **workspaces** was sourced from the community during the ellaboration of the [original **npm workspaces** RFC](https://github.com/npm/rfcs/pull/103).

## Detailed Explanation

Following up from the original **npm workspaces** RFC are the following command-related changes:

- A subset of the standard **npm commands** needs to be made aware of **npm workspaces**
- It's desired to have a standard way to filter out **workspaces** in which to run specific **npm commands**

## Rationale and Alternatives

The ability to run **npm commands** across defined **workspaces** is essential to successfully manage a **npm workspaces** workflow. That makes the following alternatives much less desireable:

- Do not implement further support to manage running commands across **workspaces**
- Implement only _some_ of the implementation changes described in this document

**Note:** Alternative ways of executing each of the described features are defined along with their implementation details below.

## Implementation

### 1. Run commands across all child packages

Make **npm cli** subcommands **npm workspaces**-aware, so that running a command tries to run it within all **workspaces** as long as a **workspaces configuration** field is properly defined in `package.json`.

Only a subset of commands are to be supported:
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved

- `fund` List funding info for all **workspaces**
- `ls` List all packages including **workspaces**
- `outdated` List outdated **dependencies** including **workspaces** and its **dependencies**
- `run-script` Run arbitrary **scripts** in all **workspaces**, skip any **workspace** that does not have a targetting **script**
Copy link

@jamiebuilds jamiebuilds Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would highly suggest a number of different options to configure the parallelization of run across workspaces

  • --parallel spawn scripts across different packages in parallel processes (default to physical cores)
  • --parallel=<count> spawn <count> processes (as much as possible)
  • --parallel-nodes split workload automatically across CI Nodes
  • --parallel-order=dependency-graph limit parallelization by workspace cross-dependency graph, so that scripts only run on packages once all other packages have been run (see task-graph-runner for example)
  • --parallel-order=any Run scripts on packages in any order

Workspace-based monorepos have a huge need to optimize their different scripts and these options go a huge way to solve for different use cases.

I have even more options in this package: https://github.com/jamiebuilds/workspaces-run

ruyadorno marked this conversation as resolved.
Show resolved Hide resolved
- `rebuild` Rebuild all **workspaces**
- `restart`
- `start`
- `stop`
- `test` Run test **scripts** in all **workspaces**
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved
- `update` Updates a **dependency** across the entire installation tree, including **workspaces**
- `view` View registry info, also including **workspaces**

#### Test example:

```
├── package.json { "name": "foo", "workspaces": ["dep-a", "dep-b"] }
├── dep-a
│ └── package.json { "version": "1.0.0" }
└── dep-b
└── package.json { "version": "1.3.1" }

$ npm test

> foo@1.0.0 test /Users/username/foo
> echo "Error: no test specified" && exit 1

Error: no test specified
npm ERR! Test failed. See above for more details.

> dep-a@1.0.0 test /Users/username/foo/dep-a
> done

> dep-b@1.3.1 test /Users/username/foo/dep-b
> done
```

**NOTE:** `publish` and `version` are very special cases in a **npm workspaces** setup and are outside of the scope of this RFC. These features will live in userland in the form of popular modules such as [Lerna](https://lerna.js.org/) for the foreseeable future.

### 2. Filter a subset of workspaces

Rather than defining a strict UX, the current proposal is to adhere to a `workspace` command prefix in the form of either: `npm workspace <workspace_name> <command>` OR `npm ws <workspace_name> <command>` OR an even shorter shortcut using a symbol `npm :<workspace_name> <command>`, for example running `test` in a specific **workspace** named `foo` with these different ideas:
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved

- `npm workspace foo test`
- `npm ws foo test`
- `npm :foo test`

Leaving space to gather feedback from the community on whether to introduce one of this proposed commands or even introducing some novel interface.
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved

#### Install dependency example:

Add `tap` as a **dependency** of a **workspace** named `foo`:

```sh
npm workspace foo install tap
npm ws foo install tap
npm :foo install tap
```

### 3. Grouping/Aliasing workspaces

One proposed feature that was surfaced during our brainstorm sessions is the ability for users to create arbitrary groups of **workspaces** defined via **workspaces configuration**, e.g:

Given a **npm workspace** setup with the following file structure:

```
/root
├── core
│ ├── foo
│ ├── bar
│ └── util
└── plugins
├── helpers
├── lorem
└── ipsum
```

It's possible to define a **groups** property within your **workspaces configuration** that will allow defining named sets of **workspaces** that can be later on used to run commands, e.g:

```
{
"name": "workspace-example",
"version": "1.0.0",
"workspaces": {
"groups": {
"core": ["core/*"], // accepts fs location
"plugins": ["lorem", "ipsum"], // also accepts workspaces names
Copy link

@mikestopcontinues mikestopcontinues Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does workspace name === subpackage name? There might be some ambiguity between directory structure and package names, so maybe, everywhere it's specified, locations should be relative paths (./packages/loren) and package names regular strings? This would go for CLI args as well as aliases. Either should be able to use globs in the context of aliases.

"common": ["util", "helpers"]
},
"packages": [
"core/*",
"plugins/*"
]
}
}
```

Following up with the previous configuration example, it would be possible to run tests across all **workspaces** from the `plugins` group, as following:

```
// one of these:
$ npm workspace plugins test
$ npm ws plugins test
$ npm :plugins test
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved

> lorem@1.0.0 test /root/plugins/lorem
> done

> ipsum@1.0.0 test /root/plugins/ipsum
> done
```

Further example, install a **peer dependency** across a group of packages named `core`:

```
$ npm workspace core install react@16 --save-peer
$ npm ws core install react@16 --save-peer
$ npm :core install react@16 --save-peer
```

## Prior Art

#### Previously

- [npm workspaces](https://github.com/npm/rfcs/blob/de8d71c0453f5cf443d3ef2f47e313f12dd6aaf9/accepted/0000-workspaces.md)

#### Filtering a subset of workspaces

- [lerna filter-options](https://www.npmjs.com/package/@lerna/filter-options)
- [Yarn v1 workspace](https://classic.yarnpkg.com/en/docs/cli/workspace)
- [Yarn v2 foreach include/exclude](https://yarnpkg.com/cli/workspaces/foreach)
- [pnpm recursive --filter](https://pnpm.js.org/en/cli/recursive#filter-lt-package_selector)
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved

## Dictionary

During the discussions around this RFC it was brought up to our attention that a lot of the vocabulary surrounding what the larger JavaScript community understands as "workspaces" can be confusing, for the sake of keeping the discussion as productive as possible we're taking the extra step of documenting what each of the terms used here means:

- **npm cli**: The [npm cli](https://github.com/npm/cli/) :wink:
- **npm commands**: The set of **npm cli** [commands](https://docs.npmjs.com/cli-documentation/)
- **npm workspaces**: The feature name, meaning the ability to the **npm cli** to support a better workflow for working with multiple packages.
- **workspaces**: A set of **workspace**s.
- **workspace**: A nested package within the **Top-level workspace** file system that is explicitly defined as such via **workspaces configuration**.
- **Top-level workspace**: The root level package that contains a **workspaces configuration** defining **workspaces**.
- **workspaces configuration**: The blob of json configuration defined within `package.json` that declares where to find **workspaces** for this **Top-level workspace** package.
- **dependency**: A package that is depended upon by another given package.
- **[peer dependency](https://docs.npmjs.com/files/package.json#peerdependencies)**: A special **dependency** relationship between packages.
- **dependent**: A package which depends on another given package.
- **symlink**: A [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) between files.
- **[globs](https://en.wikipedia.org/wiki/Glob_(programming))**: String patterns that specifies sets of filenames with special characters.
- **[Arborist](https://github.com/npm/arborist)**: The npm@7 install library
- **hoisting packages**: Bringing packages up a level in the context of an installation tree.

## Unresolved Questions and Bikeshedding

TBD