Using TS / ava 4 / ESM; entire app seems to be getting compiled separately for each test, what am I doing wrong?

[nicholas-ochoa]

I'm converting a large monorepo from Jest (tests have been broken for about six months) to Ava 4 rc1. For the most part it has been relatively smooth sailing once I got things working. Overall I'm fairly happy with ava and it seems to work great, but as I converted more and more tests over I noticed the compilation time was growing pretty excessively. The application that I'm converting takes about 6-7 seconds to build using rollup on my local dev desktop (Intel Core i9-10900 with 10 cores (20 w/ HT) and 32gb RAM). When I execute individual tests I'm seeing pretty similar compile times (using esbuild as noted below in the config).

When I run all tests together (currently 16 separate test files) I'm seeing 24 instances of esbuild spawn that are (I'm assuming) compiling my application 24 separate times before the tests are executed.

(Note: the CPU percentage displayed in the 3rd column is representative of overall CPU usage I believe, a process using 100% of a single CPU core is usually represented as using "5.00" percent of the CPU. I'm not certain why some of these are using more than that, but my overall system CPU usage is at 100% for the duration of the test execution)

This takes a good 45-60 seconds at this point and every file I add increases this by 5-6 seconds. I'm assuming that I'm doing something wrong, but honestly can't figure out what. I looked in to the @ava/typescript plugin but could not get it working - although I have TypeScript working without using that plugin so wasn't sure if it was needed.

I've considered refactoring the tests to use or call the compiled JS output of my application but haven't gone that route since it would be a pretty significant refactor. Currently each test file imports the main() function of the application and spawns its own instance on a random port (express based API app).

Aside from the issue I'm mentioning (which I've worked around by adding the timeout: 5m config option) ava is working well and doing what I expect. What can I add / change or configure to reduce the amount of compilation that is needed when running tests?

One last thing to note, I have tried to mess with the concurrency and workerThreads options. Neither seems to really solve the problem: running as child processes doesn't result in much difference besides 24 node.exe instances spawning each with an esbuild.exe child. Changing the concurrency to 1, 2, 4 or 10 didn't do much besides reduce the number of esbuild instances spawned (as expected, of course). No matter the setting the tests would take about the same amount of time (+/- 5 seconds or so) to execute to completion.

Versions being used:

• TypeScript 4.5.2
• Node.js 14.18.2
• ava 4.0.0-rc.1

ava configuration from package.json:

  "ava": {
    "failFast": false,
    "timeout": "5m",
    "extensions": {
      "ts": "module"
    },
    "nodeArguments": [
      "--no-warnings",
      "--experimental-loader=esbuild-node-loader",
      "--experimental-specifier-resolution=node"
    ]
  }

Here is an example test file (login.test.ts). @main and @util are imports from the main application being tested (aliased using TS paths config). I've excluded a few tests for brevity:

import test, { ExecutionContext } from 'ava';
import { helpers, TestContext } from '@util';
import { main } from '@main';

test.before(async (t: ExecutionContext<TestContext>) => {
  t.context = await helpers.init(main);
});

test.after.always(async (t: ExecutionContext<TestContext>) => {
  await helpers.shutdown(t.context);
});

test(`successful login`, async (t: ExecutionContext<TestContext>) => {
  const res = await t.context.request.post<any>('auth/#', {
    json: { username: 'synapse', password: 'synapse' },
  });

  t.is(res.statusCode, 200);
  t.is(res.body.username, 'SYNAPSE');
});

test(`return 400 error if no password provided`, async (t: ExecutionContext<TestContext>) => {
  const res = await t.context.request.post<any>('auth/#', {
    json: { username: 'synapse' },
  });

  t.is(res.statusCode, 400);
  t.is(res.body.error.type, 'validation');
  t.is(res.body.error.message, '"value" contains [username] without its required peers [password]');
});

test(`should return 400 if username greater than 12 characters`, async (t: ExecutionContext<TestContext>) => {
  const res = await t.context.request.post<any>('auth/#', {
    json: { username: '1234567890123' },
  });

  t.is(res.statusCode, 400);
  t.is(res.body.error.type, 'validation');
  t.is(res.body.error.message, '"username" length must be less than or equal to 12 characters long');
});

Finally, here are the tsconfig(s) used (the first one extends the second):

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "outDir": "./dist",
    "rootDir": ".",
    "baseUrl": ".",
    "paths": {
      "@container": ["src/container.ts"],
      "@interfaces": ["src/interfaces.ts"],
      "@main": ["src/main.ts"],
      "@util": ["src/utils/index.ts"],
      "@helpers": ["test/helpers/index.ts"]
    }
  },
  "include": ["src/**/*.ts"]
}

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "alwaysStrict": true,
    "allowJs": true,
    "composite": false,
    "incremental": false,
    "declaration": true,
    "esModuleInterop": true,
    "lib": ["ESNext"],
    "module": "ESNext",
    "moduleResolution": "node",
    "noImplicitAny": true,
    "noImplicitThis": true,
    "preserveSymlinks": true,
    "resolveJsonModule": true,
    "sourceMap": true,
    "strictBindCallApply": true,
    "strictFunctionTypes": true,
    "strictNullChecks": false,
    "strictPropertyInitialization": false,
    "target": "es2020",
    "typeRoots": ["node_modules/@types"]
  },
  "exclude": ["dist", "node_modules"]
}

[novemberborn]

This is the culprit: "--experimental-loader=esbuild-node-loader",

Every test file runs in a separate worker thread, so modules have to be loaded and compiled in each thread. It'll be much better to precompile using tsc directly. @ava/typescript can help you load the built test files while still letting you reference them by their TypeScript source location.

[nicholas-ochoa]

I've updated my configuration as suggested and added the @ava/typescript package. After setting compile: "tsc" and adding my tests to the tsconfig.json include property I'm seeing some progress, but the tests are still failing.

ava config (package.json relevant bits):

  ...
  "type": "module",
  ...
  "scripts": {
    "test": "ava",
    ...
  },
  ...
  "devDependencies": {
    "@ava/typescript": "^3.0.1",
    "ava": "^4.0.0-rc.1",
    ...
  },
  ...
  "ava": {
    "failFast": false,
    "timeout": "5m",
    "typescript": {
      "rewritePaths": {
        "src/": "dist/"
      },
      "compile": "tsc"
    }
  }

I've tried adding extensions: [] to the root ava config, as well as to the ava.typescript config with 'js', 'ts', or both - no matter what I seem to get the same error.

tsconfig.json:

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "outDir": "./dist",
    "rootDir": ".",
    "baseUrl": ".",
    "paths": {
      "@container": ["src/container.ts"],
      "@interfaces": ["src/interfaces.ts"],
      "@main": ["src/main.ts"],
      "@util": ["src/utils/index.ts"]
    }
  },
  "include": ["src/**/*.ts", "test/**/*.ts"]
}

The files are getting compiled correctly, I am seeing a src and test folder in dist that contains the app/tests compiled correctly and in ES module format. I also tried adding module: "commonjs" to my tsconfig.json to see if that would make a difference, but it did not.

Just for additional reference, here is an overview of how my code is structured in folders (including the output in dist):

[nicholas-ochoa]

I was able to get this working by changing a few things. I added tsc-alias as a dependency and used that to fix path imports after compiling using tsc (I couldn't use @ava/typescript for this due to the need to fix path aliases, if you're not using path aliases this step is unnecessary). Finally, I had to point ava at the compiled tests in dist/test instead of the TypeScript files in test.

  "scripts": {
    "test": "tsc && tsc-alias && ava"
  },
  "devDependencies": {
    "tsc-alias": "^1.4.2"
  },
  "ava": {
    "timeout": "30s",
    "files": ["dist/test/**/*.test.js"]
  }

Not ideal, but at least it works and doesn't try to compile the app 20 times.

[novemberborn]

If you set compile: false instead of 'tsc' then you can configure to rewrite from src/ to the dist/ output.

[nicholas-ochoa]

In the end I actually was able to remove the entire ava configuration section from my package.json as it finds my compiled test files without any problems.