refactor static workers to help with parallelization & cache sharing #68546

ztanner · 2024-08-05T19:38:41Z

What

During SSG, the fetch cache isn't shared between renders of different pages within a single export worker. This means that if Page A fetches /foo and Page B also fetches /foo, they will not be deduped and instead will be fetched from origin within each worker. Additionally, we currently only prerender 1 path per worker, which doesn't allow us to take advantage of concurrency or smarter groupings.

Why

Currently the export worker is set up to handle indivdual pages. We fire off an exportPage request to all of the known paths and jest-worker handles concurrency & parallelization.

How

This refactors the export worker to accept an array of paths. For this iteration, it doesn't attempt to do any smart grouping of paths (ie ones that share a layout, etc), it just tries to create reasonably sized groups. It will chunk a minimum of 25 pages per worker, and each worker will process experimental.staticGenerationMaxConcurrency (defaults to 2, tbd on final numbers here) exports at a time.

This lets us set up the IncrementalCache inside of the worker before we process pages. As we process pages, the incremental cache will be populated with entries, and subsequent requests will leverage the cache rules therein.

This also disables the auto no-cache behavior that we introduced with Next 15 during SSG. Unless a fetch is explicitly opting out of caching, we'll attempt to cache it during SSG so another worker can re-use it.

Closes NDX-78
Closes NDX-19

ztanner · 2024-08-05T19:38:56Z

refactor static workers to help with parallelization & cache sharing #68546 👈
canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ztanner and the rest of your teammates on Graphite

ijjk · 2024-08-05T19:41:51Z

Tests Passed

ijjk · 2024-08-05T19:46:08Z

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
buildDuration	17.2s	15.7s	N/A
buildDurationCached	8.6s	7.6s	N/A
nodeModulesSize	352 MB	352 MB	⚠️ +26 kB
nextStartRea..uration (ms)	403ms	422ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
124-HASH.js gzip	37.3 kB	37.3 kB	N/A
5121a57b-HASH.js gzip	51.9 kB	51.9 kB	N/A
7480.HASH.js gzip	169 B	169 B	✓
935-HASH.js gzip	5.19 kB	5.19 kB	N/A
framework-HASH.js gzip	56.7 kB	56.7 kB	✓
main-app-HASH.js gzip	224 B	226 B	N/A
main-HASH.js gzip	32.2 kB	32.2 kB	N/A
webpack-HASH.js gzip	1.71 kB	1.71 kB	✓
Overall change	58.6 kB	58.6 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
_app-HASH.js gzip	193 B	194 B	N/A
_error-HASH.js gzip	193 B	192 B	N/A
amp-HASH.js gzip	509 B	511 B	N/A
css-HASH.js gzip	342 B	343 B	N/A
dynamic-HASH.js gzip	1.84 kB	1.84 kB	✓
edge-ssr-HASH.js gzip	265 B	266 B	N/A
head-HASH.js gzip	362 B	364 B	N/A
hooks-HASH.js gzip	393 B	393 B	✓
image-HASH.js gzip	4.4 kB	4.4 kB	✓
index-HASH.js gzip	268 B	267 B	N/A
link-HASH.js gzip	2.81 kB	2.81 kB	N/A
routerDirect..HASH.js gzip	329 B	327 B	N/A
script-HASH.js gzip	395 B	396 B	N/A
withRouter-HASH.js gzip	325 B	323 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	6.74 kB	6.74 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
_buildManifest.js gzip	748 B	748 B	✓
Overall change	748 B	748 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
index.html gzip	521 B	521 B	✓
link.html gzip	536 B	538 B	N/A
withRouter.html gzip	517 B	518 B	N/A
Overall change	521 B	521 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
edge-ssr.js gzip	127 kB	127 kB	N/A
page.js gzip	168 kB	168 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
middleware-b..fest.js gzip	668 B	669 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	29.6 kB	29.6 kB	N/A
edge-runtime..pack.js gzip	1.03 kB	1.03 kB	✓
Overall change	1.03 kB	1.03 kB	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
928-experime...dev.js gzip	310 B	310 B	✓
928.runtime.dev.js gzip	301 B	301 B	✓
app-page-exp...dev.js gzip	246 kB	246 kB	N/A
app-page-exp..prod.js gzip	121 kB	121 kB	✓
app-page-tur..prod.js gzip	133 kB	133 kB	✓
app-page-tur..prod.js gzip	128 kB	128 kB	✓
app-page.run...dev.js gzip	232 kB	232 kB	N/A
app-page.run..prod.js gzip	117 kB	117 kB	✓
app-route-ex...dev.js gzip	23.1 kB	23.1 kB	N/A
app-route-ex..prod.js gzip	18.8 kB	18.8 kB	N/A
app-route-tu..prod.js gzip	18.8 kB	18.8 kB	N/A
app-route-tu..prod.js gzip	18.6 kB	18.6 kB	N/A
app-route.ru...dev.js gzip	24.4 kB	24.4 kB	N/A
app-route.ru..prod.js gzip	18.6 kB	18.6 kB	N/A
pages-api-tu..prod.js gzip	9.6 kB	9.6 kB	✓
pages-api.ru...dev.js gzip	9.87 kB	9.87 kB	✓
pages-api.ru..prod.js gzip	9.59 kB	9.59 kB	✓
pages-turbo...prod.js gzip	21.6 kB	21.6 kB	✓
pages.runtim...dev.js gzip	22.2 kB	22.2 kB	✓
pages.runtim..prod.js gzip	21.6 kB	21.6 kB	✓
server.runti..prod.js gzip	56.8 kB	56.8 kB	✓
Overall change	650 kB	650 kB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing	Change
0.pack gzip	1.41 MB	1.42 MB	⚠️ +1.96 kB
index.pack gzip	122 kB	120 kB	N/A
Overall change	1.41 MB	1.42 MB	⚠️ +1.96 kB

Diff details

Diff for page.js

Diff too large to display

Diff for middleware.js

Diff too large to display

Diff for edge-ssr.js

Diff too large to display

Diff for image-HASH.js

@@ -1,7 +1,7 @@
 (self["webpackChunk_N_E"] = self["webpackChunk_N_E"] || []).push([
   [8358],
   {
-    /***/ 8316: /***/ (
+    /***/ 8834: /***/ (
       __unused_webpack_module,
       __unused_webpack_exports,
       __webpack_require__
@@ -9,7 +9,7 @@
       (window.__NEXT_P = window.__NEXT_P || []).push([
         "/image",
         function () {
-          return __webpack_require__(4765);
+          return __webpack_require__(5780);
         },
       ]);
       if (false) {
@@ -18,7 +18,7 @@
       /***/
     },
 
-    /***/ 1112: /***/ (module, exports, __webpack_require__) => {
+    /***/ 7481: /***/ (module, exports, __webpack_require__) => {
       "use strict";
       /* __next_internal_client_entry_do_not_use__  cjs */
       Object.defineProperty(exports, "__esModule", {
@@ -40,17 +40,17 @@
         __webpack_require__(5596)
       );
       const _head = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(9382)
+        __webpack_require__(9639)
       );
-      const _getimgprops = __webpack_require__(2097);
-      const _imageconfig = __webpack_require__(8714);
-      const _imageconfigcontextsharedruntime = __webpack_require__(1327);
-      const _warnonce = __webpack_require__(6055);
-      const _routercontextsharedruntime = __webpack_require__(8275);
+      const _getimgprops = __webpack_require__(1607);
+      const _imageconfig = __webpack_require__(2867);
+      const _imageconfigcontextsharedruntime = __webpack_require__(1015);
+      const _warnonce = __webpack_require__(6694);
+      const _routercontextsharedruntime = __webpack_require__(9562);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(6067)
+        __webpack_require__(3463)
       );
-      const _usemergedref = __webpack_require__(1261);
+      const _usemergedref = __webpack_require__(1057);
       // This is replaced by webpack define plugin
       const configEnv = {
         deviceSizes: [640, 750, 828, 1080, 1200, 1920, 2048, 3840],
@@ -371,7 +371,7 @@
       /***/
     },
 
-    /***/ 1261: /***/ (module, exports, __webpack_require__) => {
+    /***/ 1057: /***/ (module, exports, __webpack_require__) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -440,7 +440,7 @@
       /***/
     },
 
-    /***/ 2097: /***/ (
+    /***/ 1607: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -456,9 +456,9 @@
           return getImgProps;
         },
       });
-      const _warnonce = __webpack_require__(6055);
-      const _imageblursvg = __webpack_require__(9254);
-      const _imageconfig = __webpack_require__(8714);
+      const _warnonce = __webpack_require__(6694);
+      const _imageblursvg = __webpack_require__(9866);
+      const _imageconfig = __webpack_require__(2867);
       const VALID_LOADING_VALUES =
         /* unused pure expression or super */ null && [
           "lazy",
@@ -830,7 +830,7 @@
       /***/
     },
 
-    /***/ 9254: /***/ (__unused_webpack_module, exports) => {
+    /***/ 9866: /***/ (__unused_webpack_module, exports) => {
       "use strict";
       /**
        * A shared function, used on both client and server, to generate a SVG blur placeholder.
@@ -885,7 +885,7 @@
       /***/
     },
 
-    /***/ 9331: /***/ (
+    /***/ 3484: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -912,10 +912,10 @@
         },
       });
       const _interop_require_default = __webpack_require__(4345);
-      const _getimgprops = __webpack_require__(2097);
-      const _imagecomponent = __webpack_require__(1112);
+      const _getimgprops = __webpack_require__(1607);
+      const _imagecomponent = __webpack_require__(7481);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(6067)
+        __webpack_require__(3463)
       );
       function getImageProps(imgProps) {
         const { props } = (0, _getimgprops.getImgProps)(imgProps, {
@@ -947,7 +947,7 @@
       /***/
     },
 
-    /***/ 6067: /***/ (__unused_webpack_module, exports) => {
+    /***/ 3463: /***/ (__unused_webpack_module, exports) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -982,7 +982,7 @@
       /***/
     },
 
-    /***/ 4765: /***/ (
+    /***/ 5780: /***/ (
       __unused_webpack_module,
       __webpack_exports__,
       __webpack_require__
@@ -999,8 +999,8 @@
 
       // EXTERNAL MODULE: ./node_modules/.pnpm/react@19.0.0-rc-06d0b89e-20240801/node_modules/react/jsx-runtime.js
       var jsx_runtime = __webpack_require__(3262);
-      // EXTERNAL MODULE: ./node_modules/.pnpm/next@file+..+main-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-06d0b89e-20240801_re_432dyi3er6pnalgpduyz2i6axm/node_modules/next/image.js
-      var next_image = __webpack_require__(2692);
+      // EXTERNAL MODULE: ./node_modules/.pnpm/next@file+..+diff-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-06d0b89e-20240801_re_wusbvjw3sleq6d7g7gklhdw6bm/node_modules/next/image.js
+      var next_image = __webpack_require__(8565);
       var image_default = /*#__PURE__*/ __webpack_require__.n(next_image); // CONCATENATED MODULE: ./pages/nextjs.png
       /* harmony default export */ const nextjs = {
         src: "/_next/static/media/nextjs.cae0b805.png",
@@ -1030,12 +1030,12 @@
       /***/
     },
 
-    /***/ 2692: /***/ (
+    /***/ 8565: /***/ (
       module,
       __unused_webpack_exports,
       __webpack_require__
     ) => {
-      module.exports = __webpack_require__(9331);
+      module.exports = __webpack_require__(3484);
 
       /***/
     },
@@ -1045,7 +1045,7 @@
     /******/ var __webpack_exec__ = (moduleId) =>
       __webpack_require__((__webpack_require__.s = moduleId));
     /******/ __webpack_require__.O(0, [2888, 9774, 179], () =>
-      __webpack_exec__(8316)
+      __webpack_exec__(8834)
     );
     /******/ var __webpack_exports__ = __webpack_require__.O();
     /******/ _N_E = __webpack_exports__;

Diff for 124-HASH.js

Diff too large to display

Diff for main-HASH.js

Diff too large to display

Diff for app-route-ex..ntime.dev.js

Diff too large to display

Diff for app-route-ex..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route.runtime.dev.js

Diff too large to display

Diff for app-route.ru..time.prod.js

Diff too large to display

Commit: ab786c5

test/e2e/app-dir/app-static/app-static.test.ts

packages/next/src/export/worker.ts

packages/next/src/export/index.ts

These are technically different kinds of errors and results in the worker importing everything in `build`, which was an unintentional side effect of #68546.

ijjk added created-by: Next.js team PRs by the Next.js team. tests type: next labels Aug 5, 2024

ztanner force-pushed the 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing branch 7 times, most recently from 4341d84 to 8fb51ca Compare August 6, 2024 16:40

ztanner requested a review from ijjk August 7, 2024 01:44

ztanner marked this pull request as ready for review August 7, 2024 01:44

ztanner force-pushed the 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing branch from 8fb51ca to 1a57809 Compare August 7, 2024 02:48

ijjk reviewed Aug 7, 2024

View reviewed changes

test/e2e/app-dir/app-static/app-static.test.ts Show resolved Hide resolved

ijjk reviewed Aug 7, 2024

View reviewed changes

packages/next/src/export/worker.ts Show resolved Hide resolved

ijjk reviewed Aug 7, 2024

View reviewed changes

packages/next/src/export/index.ts Outdated Show resolved Hide resolved

ztanner added 3 commits August 7, 2024 08:12

refactor static workers to help with parallelization & cache sharing

db4c16f

limit concurrency

4dca158

simplify chunking logic

285f0b4

ijjk approved these changes Aug 7, 2024

View reviewed changes

add additional config & tweak concurrency default

ab786c5

ztanner force-pushed the 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing branch from 1a57809 to ab786c5 Compare August 7, 2024 16:35

ztanner merged commit d365ff8 into canary Aug 7, 2024
108 checks passed

ztanner deleted the 08-05-refactor_static_workers_to_help_with_parallelization_cache_sharing branch August 7, 2024 21:13

ztanner mentioned this pull request Aug 13, 2024

remove circular ExportError import from SSG worker #68858

Merged

github-actions bot added the locked label Aug 22, 2024

github-actions bot locked as resolved and limited conversation to collaborators Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor static workers to help with parallelization & cache sharing #68546

refactor static workers to help with parallelization & cache sharing #68546

ztanner commented Aug 5, 2024 •

edited

Loading

ztanner commented Aug 5, 2024

ijjk commented Aug 5, 2024 •

edited

Loading

ijjk commented Aug 5, 2024 •

edited

Loading

refactor static workers to help with parallelization & cache sharing #68546

refactor static workers to help with parallelization & cache sharing #68546

Conversation

ztanner commented Aug 5, 2024 • edited Loading

What

Why

How

ztanner commented Aug 5, 2024

ijjk commented Aug 5, 2024 • edited Loading

Tests Passed

ijjk commented Aug 5, 2024 • edited Loading

Stats from current PR

ztanner commented Aug 5, 2024 •

edited

Loading

ijjk commented Aug 5, 2024 •

edited

Loading

ijjk commented Aug 5, 2024 •

edited

Loading