node >=17: experimental esm loader doesnt respect --experimental-specifier-resolution=node

[noahehall]

Version

17.3.0

Platform

Linux spaceship 5.13.0-22-generic #22-Ubuntu SMP Fri Nov 5 13:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

loaders, experimental node options

What steps will reproduce the bug?

• create an esm loader for .mjs files
• create two files, main.mjs which imports somefile.mjs
• run the main.mjs with experimental-specifier-resolution=node via node 17.3.0

esm loader works under the following conditions

import poop from './somefile.mjs'
import poop from '@someNamespace/somePackage''

esm loader fails under the following conditions

import poop from './somefile'

How often does it reproduce? Is there a required condition?

everyday, except sundays. thats the lords day

What is the expected behavior?

i can use experimental loader + experimental specifier resolution because Ive spent many days reading the nodejs docs, and the nodejs docs never lie

What do you see instead?

poop@spaceship:~/git/foss/nodeproto/packages/libraries/testproto
06:57 AM (test-everything *$=) 
$ pnpm repo:test

> @nodeproto/testproto@0.0.0 repo:test /home/poop/git/foss/nodeproto/packages/libraries/testproto
> NODE_OPTIONS="$npm_package_config_NODE_OPTIONS" uvu ./src "test.(m|c)?js"

(node:1377741) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
TypeError [ERR_INVALID_FILE_URL_HOST]: File URL host must be "localhost" or empty on linux
    at statSync (node:fs:1531:10)
    at fileExists (node:internal/modules/esm/resolve:260:10)
    at resolveExtensions (node:internal/modules/esm/resolve:335:9)
    at resolveExtensionsWithTryExactName (node:internal/modules/esm/resolve:322:10)
    at finalizeResolution (node:internal/modules/esm/resolve:375:16)
    at moduleResolve (node:internal/modules/esm/resolve:915:10)
    at defaultResolve (node:internal/modules/esm/resolve:1005:11)
    at ESMLoader.resolve (node:internal/modules/esm/loader:530:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:251:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:79:40)
 ELIFECYCLE  Command failed with exit code 1.

poop@spaceship:~/git/foss/nodeproto/packages/libraries/testproto
06:57 AM (test-everything *$=) 
$ 

Additional information

06:57 AM (test-everything *$=) 
$ pnpm repo:about

> @nodeproto/testproto@0.0.0 repo:about /home/poop/git/foss/nodeproto/packages/libraries/testproto
> npm run-script && npm --versions && pnpm c list && echo "NODE OPTIONS set to: $npm_package_config_NODE_OPTIONS"

npm info using npm@8.3.0
npm info using node@v17.3.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 1ms
npm timing config:load:env Completed in 1ms
npm timing config:load:file:/home/poop/git/foss/nodeproto/packages/libraries/testproto/.npmrc Completed in 0ms
npm timing config:load:project Completed in 1ms
npm timing config:load:file:/home/poop/.npmrc Completed in 0ms
npm timing config:load:user Completed in 1ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 1ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 1ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:display Completed in 6ms
npm timing npm:load:logFile Completed in 4ms
npm timing npm:load:timers Completed in 0ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load Completed in 21ms
Scripts available in @nodeproto/testproto@0.0.0 via `npm run-script`:
  build
    pnpm repo:rm:dist && pnpm repo:cp:cjs && swc $npm_package_config_DIRS_TO_TRANSPILE -d $npm_package_config_PATH_DIST --config-file ./node_modules/@nodeproto/configproto/src/swc/node.swcrc --copy-files
  proto:bin
    pnpm -r exec $*
  proto:script
    pnpm -r $*
  proto:slice
    NODE_OPTIONS="$npm_package_config_NODE_OPTIONS" pnpm exec ultra -r --filter
  proto
    pnpm exec ultra -r
  repo:about
    npm run-script && npm --versions && pnpm c list && echo "NODE OPTIONS set to: $npm_package_config_NODE_OPTIONS"
  repo:cp:browserslist
    rm -f .browserslistrc && cp ./node_modules/@nodeproto/configproto/.browserslistrc .
  repo:cp:cjs
    rm -f ./dist/package.json && mkdir -p dist && cp ./node_modules/@nodeproto/configproto/src/node/cjs.json ./dist/package.json
  repo:cp:configproto
    pnpm repo:cp:flow; pnpm repo:cp:cjs; pnpm repo:cp:browserslist
  repo:cp:flow
    rm -f .flowconfig && cp ./node_modules/@nodeproto/configproto/.flowconfig .
  repo:deps
    ultra --info
  repo:eslint:fix
    pnpm repo:eslint -- --fix
  repo:eslint
    eslint './**/*'
  repo:flowtyped:install
    flow-typed install
  repo:nuke
    pnpm repo:rm:dist; pnpm repo:rm:nodemodules
  repo:rm:dist
    rm -rf dist/*
  repo:rm:nodemodules
    rm -rf node_modules/*
  repo:scripts:v
    npm run-script
  repo:scripts
    ultra --list
  repo:test:file
    node --experimental-specifier-resolution=node --experimental-import-meta-resolve $*
  repo:test:help
    uvu --help
  repo:test
    NODE_OPTIONS="$npm_package_config_NODE_OPTIONS" uvu ./src "test.(m|c)?js"
  repo:testing
    pnpm repo:test; chokidar "src/**/*(?:.test)?.(m|c)?js" -c "pnpm repo:test"

npm timing command:run-script Completed in 9ms
npm timing npm Completed in 117ms
npm info ok 
npm info using npm@8.3.0
npm info using node@v17.3.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/lib/node_modules/npm/npmrc Completed in 0ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 1ms
npm timing config:load:env Completed in 1ms
npm timing config:load:file:/home/poop/git/foss/nodeproto/packages/libraries/testproto/.npmrc Completed in 0ms
npm timing config:load:project Completed in 2ms
npm timing config:load:file:/home/poop/.npmrc Completed in 0ms
npm timing config:load:user Completed in 0ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 1ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 0ms
npm timing config:load:flatten Completed in 3ms
npm timing npm:load:display Completed in 7ms
npm timing npm:load:logFile Completed in 4ms
npm timing npm:load:timers Completed in 0ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load Completed in 22ms
npm timing config:load:flatten Completed in 1ms
{
  '@nodeproto/testproto': '0.0.0',
  npm: '8.3.0',
  node: '17.3.0',
  v8: '9.6.180.15-node.12',
  uv: '1.42.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '102',
  nghttp2: '1.45.1',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '3.0.1+quic',
  cldr: '40.0',
  icu: '70.1',
  tz: '2021a3',
  unicode: '14.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}
npm timing command:version Completed in 8ms
npm timing npm Completed in 116ms
npm info ok 
npm info using npm@8.3.0
npm info using node@v17.3.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 1ms
npm timing config:load:env Completed in 1ms
npm timing config:load:file:/home/poop/git/foss/nodeproto/packages/libraries/testproto/.npmrc Completed in 0ms
npm timing config:load:project Completed in 1ms
npm timing config:load:file:/home/poop/.npmrc Completed in 0ms
npm timing config:load:user Completed in 0ms
npm timing config:load:file:/opt/nodejs/node-v17.3.0-linux-x64/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 1ms
npm timing config:load:validate Completed in 1ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 0ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:display Completed in 7ms
npm timing npm:load:logFile Completed in 4ms
npm timing npm:load:timers Completed in 0ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load Completed in 21ms
; "env" config from environment

_noahehall-registry = (protected) 
argv = "{\"remain\":[\"run\",\"repo:about\"],\"cooked\":[\"run\",\"repo:about\"],\"original\":[\"run\",\"repo:about\"]}" 
cache-dir = "/var/.nodeproto/pnpm" 
child-concurrency = "2" 
color = true 
enable-modules-dir = true 
fetch-retry-maxtimeout = 60000 
fetch-retry-mintimeout = 10000 
fetch-timeout = 60000 
global-bin-dir = "/var/.nodeproto/pnpm-global/bin" 
global-dir = "/var/.nodeproto/pnpm-global" 
hoist = true 
hoist-pattern = "['*']" 
link-workspace-packages = true 
lockfile = true 
loglevel = "info" 
modules-cache-max-age = "2880" 
modules-dir = "node_modules" 
node-gyp = "/opt/nodejs/node-v17.3.0-linux-x64/pnpm-global/5/node_modules/.pnpm/pnpm@6.24.2/node_modules/pnpm/dist/node_modules/node-gyp/bin/node-gyp.js" 
package-import-method = "copy" 
prefer-frozen-lockfile = true 
prefer-offline = true 
prefer-workspace-packages = true 
recursive-install = true 
registry = "https://registry.npmjs.org/" 
save-exact = true 
save-prefix = "" 
shared-workspace-lockfile = true 
state-dir = "/var/.nodeproto/pnpm" 
store-dir = "/var/.nodeproto/.pnpm-store" 
symlink = true 
tag = "latest" 
use-beta-cli = true 
user-agent = "pnpm/6.24.2 npm/? node/v17.3.0 linux x64" 
virtual-store-dir = "node_modules/.pnpm" 

; node bin location = /opt/nodejs/node-v17.3.0-linux-x64/bin/node
; cwd = /home/poop/git/foss/nodeproto/packages/libraries/testproto
; HOME = /home/poop
; Run `npm config ls -l` to show all defaults.
npm timing command:c Completed in 7ms
npm timing npm Completed in 113ms
npm info ok 
NODE OPTIONS set to: --experimental-loader="./node_modules/@nodeproto/configproto/src/node/loaders/flow.mjs" --experimental-specifier-resolution=node --experimental-json-modules --experimental-top-level-await --experimental-vm-modules --experimental-import-meta-resolve node --enable-source-maps --heapsnapshot-near-heap-limit=3 --use-largepages=on --pending-deprecation --trace-sigint --trace-warnings --trace-exit --report-uncaught-exception --report-on-fatalerror --abort-on-uncaught-exception

poop@spaceship:~/git/foss/nodeproto/packages/libraries/testproto
06:57 AM (test-everything *$=) 

[noahehall]

not sure if this matters, but here is the relevant section of my pkg json

"exports": {
    ".": {
      "import": "./src/index.mjs",
      "default": "./dist/index.js"
    },
    "./t": {
      "import": "./src/t.mjs",
      "default": "./dist/t.js"
    }
  },

this is the line in the esm loader that throws the err

if (!isFlow) return isCjs 
  ? { format, source: rawSource } 
   : defaultLoad(url, { format });  <--- this procedure

[aduh95]

/cc @nodejs/loaders

[bmeck]

Need loader source text

…

On Sat, Dec 25, 2021, 11:32 AM Antoine du Hamel ***@***.***> wrote: /cc @nodejs/loaders <https://github.com/orgs/nodejs/teams/loaders> — Reply to this email directly, view it on GitHub <#41321 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZJIYAYZOYIERHGF6KYCTUSX5ZLANCNFSM5KXX5NKA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[noahehall]

@bmeck loader source text

[noahehall]

still broken in 17.4

in /src/t.mjs it exports a number of things
in /src/t.test.mjs it imports all things under the name space t

import * as t from ./t fails
import * as t from ./t.mjs works
import * as t from @nodeproto/testproto works

console log

poop@spaceship:~/git/foss/nodeproto/packages/libraries/testproto
10:48 AM (rework-testproto *$=) 
$ pnpm repo:test

> @nodeproto/testproto@0.0.0 repo:test /home/poop/git/foss/nodeproto/packages/libraries/testproto
> NODE_OPTIONS="$npm_package_config_NODE_OPTIONS" uvu ./src "test.(m|c)?js"

(node:614409) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
    at emitExperimentalWarning (node:internal/util:224:11)
    at initializeLoader (node:internal/process/esm_loader:59:3)
    at loadESM (node:internal/process/esm_loader:87:11)
    at runMainESM (node:internal/modules/run_main:51:21)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:74:5)
    at node:internal/main/run_main_module:17:47
TypeError [ERR_INVALID_FILE_URL_HOST]: File URL host must be "localhost" or empty on linux
    at __node_internal_captureLargerStackTrace (node:internal/errors:464:5)
    at new NodeError (node:internal/errors:371:5)
    at getPathFromURLPosix (node:internal/url:1459:11)
    at fileURLToPath (node:internal/url:1482:50)
    at toPathIfFileURL (node:internal/url:1561:10)
    at __node_internal_ (node:internal/fs/utils:681:16)
    at statSync (node:fs:1531:10)
    at fileExists (node:internal/modules/esm/resolve:260:10)
    at resolveExtensions (node:internal/modules/esm/resolve:335:9)
    at resolveExtensionsWithTryExactName (node:internal/modules/esm/resolve:322:10)
    at finalizeResolution (node:internal/modules/esm/resolve:375:16)
    at moduleResolve (node:internal/modules/esm/resolve:944:10)
    at defaultResolve (node:internal/modules/esm/resolve:1041:11)
    at ESMLoader.resolve (node:internal/modules/esm/loader:530:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:251:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:79:40)
    at link (node:internal/modules/esm/module_job:78:36)
(node:614409) WARNING: Exited the environment with code 1
    at exit (node:internal/process/per_thread:194:13)
    at /home/poop/git/foss/nodeproto/node_modules/.pnpm/uvu@0.5.3/node_modules/uvu/bin.js:32:12
 ELIFECYCLE  Command failed with exit code 1.

poop@spaceship:~/git/foss/nodeproto/packages/libraries/testproto
10:49 AM (rework-testproto *$=) 
$ 

[Nick-Mazuk]

Seems to still be broken in v18.0

[GeoffreyBooth]

The plan is to remove --experimental-specifier-resolution=node once the equivalent functionality can be achieved via a custom loader without an unreasonable amount of boilerplate. See https://nodejs.org/api/esm.html#customizing-esm-specifier-resolution-algorithm.

Therefore, this issue is unlikely to be fixed unless it can be demonstrated to have wider implications than behavior under --experimental-specifier-resolution=node.

[bnoordhuis]

--experimental-specifier-resolution=node has been removed. I'll go ahead and close this.