"Could not read from file …" error thrown when using ?url and ?worker suffixes inside 3rd party module

[wojtekmaj]

Describe the bug

Getting module URL in our own code like so:

import module2Url from "module-2/index?url";

works fine, but if that very same line is placed in node_modules/module-1/index.js, build crashes:

wmaj@MacBook-Pro vite-url-suffix-bug % yarn dev          
Port 5173 is in use, trying another one...

  VITE v3.2.3  ready in 83 ms

  ➜  Local:   http://localhost:5174/
  ➜  Network: use --host to expose
✘ [ERROR] Could not read from file: /vite-url-suffix-bug/node_modules/module-2/index.js?url

    node_modules/module-1/index.js:1:23:
      1 │ import module2Url from "module-2/index?url";
        ╵                        ~~~~~~~~~~~~~~~~~~~~

/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1566
  let error = new Error(`${text}${summary}`);
              ^

Error: Build failed with 1 error:
node_modules/module-1/index.js:1:23: ERROR: Could not read from file: /vite-url-suffix-bug/node_modules/module-2/index.js?url
    at failureErrorWithLog (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1566:15)
    at /vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1022:7)
    at /vite-url-suffix-bug/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/vite-url-suffix-bug/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12) {
  errors: [
    {
      detail: undefined,
      id: '',
      location: {
        column: 23,
        file: 'node_modules/module-1/index.js',
        length: 20,
        line: 1,
        lineText: 'import module2Url from "module-2/index?url";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'Could not read from file: /vite-url-suffix-bug/node_modules/module-2/index.js?url'
    }
  ],
  warnings: []
}

Node.js v18.12.0

At the same time, I was not able to find any mention regarding these suffixed working only on limited subset of modules when reading the following page: https://vitejs.dev/guide/assets.html

Reproduction

https://github.com/wojtekmaj/vite-url-suffix-bug

Steps to reproduce

• yarn
• yarn copy-packages (to replace symlinks in node_modules with actual modules, since they come from internal workspace)
• yarn dev
• 💥

System Info

System:
    OS: macOS 13.0
    CPU: (8) arm64 Apple M1 Pro
    Memory: 457.75 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.12.0 - /usr/local/bin/node
    Yarn: 3.2.4 - /usr/local/bin/yarn
    npm: 8.19.2 - /usr/local/bin/npm
  Browsers:
    Chrome: 107.0.5304.87
    Firefox: 106.0.3
    Safari: 16.1
  npmPackages:
    vite: ^3.2.3 => 3.2.3

Used Package Manager

yarn

Logs

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[seo-rii]

Is there any update about this?

[TheMoonDawg]

In my case, I was updating to React PDF v7 and ran into this issue. What I ended up doing was including the transitive dependency / 3rd party module in our Vite config to also optimize it when using the ?url suffix. Worked like a charm!

export default defineConfig({
  ...
  optimizeDeps: { include: ['pdfjs-dist'] },
})

[Danielku15]

I dived a bit into the bug and the problem is in this line:

vite/packages/vite/src/node/plugins/workerImportMetaUrl.ts

Line 154 in e526573

file = tryFsResolve(file, fsResolveOptions) ?? file

If the file system resolve fails here, we just stay on the artificially produced path like: 'D:\Dev\myproj\src\vite-react\node_modules\.vite\deps\mylib.worker.mjs'. This leads to two problems:

1. The worker file is considered an "optimized dependency" because its file path starts with .vite\deps

   vite/packages/vite/src/node/optimizer/index.ts

   Lines 912 to 921 in f8e0791

   	function getDepsCacheDirPrefix(config: ResolvedConfig): string {
   	return normalizePath(path.resolve(config.cacheDir, 'deps'))
   	}
   	export function createIsOptimizedDepFile(
   	config: ResolvedConfig,
   	): (id: string) => boolean {
   	const depsCacheDirPrefix = getDepsCacheDirPrefix(config)
   	return (id) => id.startsWith(depsCacheDirPrefix)
   	}

2. This path is then encoded and decoded to URLs and used by modules but will end up with files which cannot be found.

Hence a fix would mean: resolve at this location the correct path to the worker file of the 3rd party module in the node_modules directory.

It seems a code bit like this will work:

file = tryFsResolve(file, fsResolveOptions) ?? tryOptimizedDepResolve(url, id) ?? file;


function tryOptimizedDepResolve(url, depId, fsResolveOptions) {
	if (depsOptimizer?.isOptimizedDepFile(depId)) {
		const depFile = cleanUrl(depId);
		const info = optimizedDepInfoFromFile(depsOptimizer.metadata, depFile);
		const depSrc = info.src;
		if (depSrc) {
			const resolvedFile = path.resolve(path.dirname(depSrc), url)
            return tryFsResolve(file, fsResolveOptions);
		}
	}
	
	return undefined;
}

It is a similar flow like the existing tryOptimizedResolve but simplified.

I'll check if I can make a change to Vite and test it locally.

[Danielku15]

I prepared a fix which works nicely for my library at #16418
Would be great if some more people could give it a test. To test it locally:

1. Clone and built vite from https://github.com/Danielku15/vite/tree/fix/workers-in-3rd-party-modules (pnpm install && pnpm build)
2. Copy the local vite build to your project where you use vite with a 3rd party library.
3. Run and test your application and check if the worker loads.

[timephy]

Running into the same issue, would love to see this fixed..