[rush-lib] Support pnpm lockfile v9

[fzxen]

Summary

pnpm lockfile v9 have some breaking changes on the lockfile format. Rush cannot parse pnpm lockfile v9 correctly with latest version.
After i execute rush install or rush update with pnpm v9, the content of .rush/temp/shrinkwrap-deps.json is not correct. It may break rush cache system.
The expected output should correctly display the hash of each dependency., but actually:

Details

pnpm have some breaking changes on lockfile v9 format.

1. The package field has been divided into two parts, package and snapshot.
2. In non-workspace mode, the top-level dependencies field is moved into importers['.'].
3. specifier is not same as lockfile v6

slolution to 1:

Rush will load the lockfile and parse the information in the lockfile by itself.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 318 to 347 in 3c6117e

	public static loadFromFile(
	shrinkwrapYamlFilePath: string,
	{ withCaching }: ILoadFromFileOptions = {}
	): PnpmShrinkwrapFile | undefined {
	let loaded: PnpmShrinkwrapFile | undefined;
	if (withCaching) {
	loaded = PnpmShrinkwrapFile._cacheByLockfilePath.get(shrinkwrapYamlFilePath);
	}
	// TODO: Promisify this
	loaded ??= (() => {
	try {
	const shrinkwrapContent: string = FileSystem.readFile(shrinkwrapYamlFilePath);
	return PnpmShrinkwrapFile.loadFromString(shrinkwrapContent);
	} catch (error) {
	if (FileSystem.isNotExistError(error as Error)) {
	return undefined; // file does not exist
	}
	throw new Error(`Error reading "${shrinkwrapYamlFilePath}":\n ${(error as Error).message}`);
	}
	})();
	PnpmShrinkwrapFile._cacheByLockfilePath.set(shrinkwrapYamlFilePath, loaded);
	return loaded;
	}
	public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile {
	const parsedData: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent);
	return new PnpmShrinkwrapFile(parsedData);
	}

I use the @pnpm/lockfile.fs library to load the pnpm lockfile. Ensure that the parsing logic is consistent with pnpm. The readWantedLockfile method will automatically merge the snapshot field information into the package field. If we need to support pnpm's git branch lockfile, only the parameters of the method need to be adjusted.

if (lockfileVersion >= ShrinkwrapFileMajorVersion.V9) {
  const { readWantedLockfile } = await import('@pnpm/lockfile.fs');
  const lockfile: IPnpmShrinkwrapYaml | null = await readWantedLockfile(
    path.dirname(shrinkwrapYamlFilePath),
    {
      ignoreIncompatible: false
      // TODO support git branch lockfile
      // useGitBranchLockfile: false,
      // mergeGitBranchLockfiles: false,
    }
  );
}

loaded = new PnpmShrinkwrapFile(shrinkwrapFileJson);

solution to 2:

In the PnpmShrinkwrapFile.loadFile method, use importers['.'].dependencies instead of top-level dependencies.

// if (lockfileVersion >= ShrinkwrapFileMajorVersion.V9) {
// ...
if (lockfile) {
    lockfile.dependencies = lockfile.importers['.' as ProjectId]?.dependencies;
    shrinkwrapFileJson = lockfile;
  }
// }

solution to 3:

rush try to parse an encoded pnpm dependency key in parsePnpmDependencyKey method. However, the logic here is no longer applicable to lockfile v9. 例如，lockfile v9 will not add a / prefix in the specifier field.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 188 to 198 in 3c6117e

	// Example: "path.pkgs.visualstudio.com/@scope/depame/1.4.0" --> 0="@scope/depame" 1="1.4.0"
	// Example: "/isarray/2.0.1" --> 0="isarray" 1="2.0.1"
	// Example: "/sinon-chai/2.8.0/chai@3.5.0+sinon@1.17.7" --> 0="sinon-chai" 1="2.8.0/chai@3.5.0+sinon@1.17.7"
	// Example: "/typescript@5.1.6" --> 0=typescript 1="5.1.6"
	// Example: 1.2.3_peer-dependency@.4.5.6 --> no match
	// Example: 1.2.3_@scope+peer-dependency@.4.5.6 --> no match
	// Example: 1.2.3(peer-dependency@.4.5.6) --> no match
	// Example: 1.2.3(@scope/peer-dependency@.4.5.6) --> no match
	const packageNameMatch: RegExpMatchArray | null = /^[^\/(]*\/((?:@[^\/(]+\/)?[^\/(]+)[\/@](.*)$/.exec(
	dependencyKey
	);

Summary of changes to specifier in lockfile v9：

1. remove prefix '/‘: '/@babel/preset-env@7.26.0(@babel/core@7.26.0)' -> @babel/preset-env@7.26.0(@babel/core@7.26.0)
2. URLs specifier always prefix with https:.
3. it will prefix with <PACKAGE_NAME>@ if resolved package name is not same as package.json

category	specifier	lockfilev6 version	lockfilev9 version
regular	"@babel/plugin-preset-env": "^7.26.0"	7.26.0(@babel/core@7.26.0)	7.26.0(@babel/core@7.26.0)
URLs	"pad-left": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"pad-left": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"pad-left": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"pad-left": "git+ssh://git@github.com:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
file:	"project1": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	file:../pnpm_no_workspace/projects/project1
path	"project1": "../pnpm_no_workspace/projects/project1"	link:../pnpm_no_workspace/projects/project1	link:../pnpm_no_workspace/projects/project1
Npm alias	"test-pkg": "npm:@babel/preset-env@7.26.0"	/@babel/preset-env@7.26.0(@babel/core@7.26.0)	@babel/preset-env@7.26.0(@babel/core@7.26.0)
Alias with URLs	"@scope/myDep1": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	pad-left@https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"@scope/myDep2": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	pad-left@https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"@scope/myDep3": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"@scope/myDep4": "git+ssh://git@github.com:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
Alias with file:	"test-pkg": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	project1@file:../pnpm_no_workspace/projects/project1

It is difficult to maintain those complex regular expressions in the original function. Therefore, I added a new function named as parsePnpm9DependencyKey. It will be called when the expression shrinkwrapFileMajorVersion >= 9 is true in runtime.

const result: DependencySpecifier | undefined =
        this.shrinkwrapFileMajorVersion >= ShrinkwrapFileMajorVersion.V9
          ? parsePnpm9DependencyKey(dependencyName, pnpmDependencyKey)
          : parsePnpmDependencyKey(dependencyName, pnpmDependencyKey);

This MR will not cause any breaking changes. But, pnpm9 and rush subspaces still cannot work together, because pnpm-sync not support pnpm9 yet. tiktok/pnpm-sync#37

I added some unit test cases in PnpmShrinkwrapFile.test.ts and ShrinkwrapFile.test.ts.

[fzxen]

@microsoft-github-policy-service agree