`ts-node` fails when ES Modules are in the dependency graph in Node.js 13+

[trusktr]

I basically detailed the issue in this comment: #155 (comment)

It's a chicken-and-egg-like problem:

• If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.
• If we change to module: 'esnext', then the errors from the previous point go away, but now the .js file that is loading ts-node and calling something like require('typescript-entry-point.ts') will have a similar issue, because the call to require('typescript-entry-point.ts') will try to load an ES Module.
• Finally, if we convert the .js file into an ES Module, we can not convert require('typescript-entry-point.ts') into import 'typescript-entry-point.ts' because now ES Modules don't handle .ts extensions (at least not out of the box, and it seems the old require hooks don't operate on these new identifiers)

At the moment, I'm sort of stuck, because I have dependencies in my dependency tree that are ES Modules.

The only workaround I can think of is to compile everything to .js files (ES Modules) and avoid to use ts-node.

I wonder if a combination of allowJs and ignore so that it compiles JS files would help. I haven't tried that yet.

[trusktr]

One way to reproduce this is to take a working application with module: 'commonjs' being passed to the register() call, make sure you're using the latest Node 13, then add an ES Module dependency (namely a .js file inside a node_modules package with type: 'module', as I believe that'll be the prime way that people will publish ES Modules from now on) on the leaf end of any branch of the dependency tree.

[trusktr]

related:

• Support .mjs output [also a TypeScript issue] #436
• Node option for experimental and module resolutions #880

Now that Node 13 is out in the wild, looks like people are starting to run into this without any flags. Eventually someone will have unpaid free time to fix it, but I think it is easier to just pre-compile everything to JS before running Node.

[jkrems]

If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.

I'm trying to understand this scenario - how did this work before? I assume existing TypeScript running via ts-node was never meant to use real ESM semantics, instead it's more similar to babel-style "using import as sugar for require". So it's de-facto CommonJS and shouldn't be loading ES modules from its dependencies (at least that should be the default behavior afaict).

Is this about existing projects breaking or about existing projects migrating from CJS-"import" to standard imports?

[jkrems]

Ah, I think I got it now: It's about running ts-node for module code. Which would involve compilation within the ES module loader, potentially registering the compiler after node is started up (as opposed to using a CLI flag). The short answer is that the APIs to allow that aren't yet in node, so I don't think ts-node can do anything about this for now. :(

[trusktr]

Not even by hooking in to the experimental load API?

[jkrems]

There's two independent holes in the API surface:

1. Being able to change the source text of something loaded from disk without breaking relative resolution. There is an open PR to address this by @GeoffreyBooth: esm: source hooks nodejs/node#30986
2. Being able to start a simple binary (ts-node) that registers a transformer and then runs user code. Right now the only way to register hooks is using a node flag (node --some-flag=ts-node main.ts). Short of forking a new node process, there's currently no API that allows to implement ts-node main.ts with native module support.

The second point is tricky because it's by design. We don't really want to allow userland code (from node's perspective) being able to switch out parts of the loading pipeline at runtime. It can lead to weird issues where the same global scope contains code expecting different module loaders and thanks to import() that's actually observable (as opposed to require).

One idea brought up by @guybedford was to have an API that can create a new global scope ("context" or "realm") with all bells and whistles, including node standard library, process global, and fresh module system. Tools like ts-node could use it to set up a new context for running the application code. But there are some downsides and questions, e.g. around garbage collection of the original "bootstrap" context and the time it would take to initialize the new context.

[EntraptaJ]

I created a proof of concept for using the current version of the --experimental-loader feature to transpile typescript files right before node loads it and then give node the now ESNext Javascript code with the SyntheticModule VM feature. It's fairly hacky right now but it made for a fun 6 hour session: https://github.com/KristianFJones/TS-ES-Node

[Jamesernator]

^ Just for above there's a new loader hook in node nightly called getSource so the following loader can be used in node nightly. Only caveats are the same as the babel typescript plugin (e.g. no const enum):

Typescript Loader

import fs from 'fs';
import path from 'path';
import babel from '@babel/core';
import urlUtils from 'url';
import babelPluginTransformTypescript from '@babel/plugin-transform-typescript';

function isURL(string) {
    try {
        new URL(string);
        return true;
    } catch {
        return false;
    }
}
const removePrivateTypes = (babel) => {
    return {
        visitor: {
            ClassPrivateProperty(context) {
                context.get('typeAnnotation')?.remove();
            },
        },
    };
}

const BABEL_OPTS = {
    plugins: [
        babelPluginTransformTypescript,
        removePrivateTypes,
    ],
    parserOpts: {
        plugins: [
            'asyncGenerators',
            'bigInt',
            'classProperties',
            'classPrivateProperties',
            'importMeta',
            'nullishCoalescingOperator',
            'numericSeparator',
            'objectRestSpread',
            'optionalCatchBinding',
            'optionalChaining',
            'topLevelAwait',
            'typescript',
        ],
    },
    sourceMaps: 'inline',
};

export async function getSource(urlString, context, getSourceDefault) {
    if (isURL(urlString)) {
        const url = new URL(urlString);
        if (url.pathname.endsWith('.js') && !fs.existsSync(url)) {
            url.pathname = url.pathname.replace(/\.js$/u, '.ts');
            const contents = await fs.promises.readFile(url, 'utf8');
            const { code: source } = await babel.transformAsync(contents, {
                ...BABEL_OPTS,
                sourceFileName: path.basename(urlUtils.fileURLToPath(url)),
            })
            return { source };
        }
    }
    return getSourceDefault(urlString, context, getSourceDefault);
}

export async function resolve(specifier, context, defaultResolve) {
    try {
        const defaultResolution = defaultResolve(specifier, context, defaultResolve);

        try {
            const url = new URL(defaultResolution.url);
            url.pathname = url.pathname.replace(/\.ts$/, '.js');
            return { url: url.href };
        } catch {
            return defaultResolution;
        }
    } catch {
        return { url: new URL(specifier, context.parentURL).href };
    }
}

[GeoffreyBooth]

Hi, I created the getSource hook. There's also a transformSource hook that was intended for things like transpilation. See the transpiler loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader

[EntraptaJ]

Look's like I know what I'm doing this weekend.

[EntraptaJ]

Been busy, here is a proof of concept using the transformSource hook. Supports relative imports of .ts and .tsx without requiring extensions. https://github.com/K-FOSS/TS-ESNode

[EntraptaJ]

Okay, made a few small changes and fixes. It's now able to handle external modules that have the TSLib import helper without requiring refactoring existing imports. The TS-ESNode loader hook can be dropped into existing projects and just work. Hopefully! I just tested it on my main Application template that uses TypeORM & TypeGraphQL with tsconfig.json set to use ESNext as target and modules and it all just works. Transpiled code is all using imports and exports with Node V14 in Module mode.

[EntraptaJ]

Removed the globby requirement. Module has been published on NPM https://www.npmjs.com/package/@k-foss/ts-esnode zero dependencies other then peer dependency on TypeScript. NPM reports unpacked size of 8.66 kB. Should hopefully be a drop in replacement for TS-Node for now.