[v2] improve how we load GraphQL query results in development/product…

…ion (aka Ludicrous Mode) (gatsbyjs#4555) * Create placeholder JSON store * Rename * Websocket placeholder * Push query results JSON over websockets * More descriptive variable name * Fix queries being overwritten * Remove eslint-disable flag * Remove junk * test require error fix for windows * dont require json data in sync-require * dont add layout data to json array multiple times * initial async loading * revert saving json directly to public for now * updated production-app to sync with prop name change in ComponentRenderer * we load json data via json-loader component in develop and not handling it with webpack import/require * hashes for json files * fix preloading, use xhr instead of fetch - for some reason can't force fetch to not create additional request, with any `cache` or `mode` configuration * dont use full paths in dataPath - remove static/d/ path and .json ext - results in smaller app bundle especially with large ammount of pages * Enable cached query results to be loaded * Don't dump all query results out to the client Instead only push results out if the data is for a path that's currently being viewed in a client. * fix preload link to json data * remove not used function * remove more not used code * Update to latest webpack/mini-css-extract-plugin * don't write new (a)sync-requires.js if components didn't change (gatsbyjs#4759) * create just one websocket client (gatsbyjs#4763) * Filter out duplicate query jobs and create secondary queue for jobs if path already has query in flight * [json-loader] Don't emit new file node until previous is finished processing (gatsbyjs#4785) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * [json-loader] Only log file events if we're past bootstrap (gatsbyjs#4826) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * Only log file events if we're past bootstrap * [json-loader] dont recompile on data change - part 2 (gatsbyjs#4837) * prevent adding duplicate redirects * don't write new `redirects.json` if redirects didn't change prevents webpack recompilation on data change * [json-loader] develop - reading results from file improvments (gatsbyjs#4850) * dont emit results for layouts * [develop] store query results in memory, read json data from file only if we don't have it stored yet (we didn't run this query, but results are cached) * Add query prioritization based on what page(s) user(s) are on Query running is sadly not very ludicrous right now on gatsbyjs.org — not sure why — each markdown file change causes ~20 queries to run but even with prioritizing the active page's query, it's still ~2 seconds before the page updates. This sort of thing will be much easier to debug with tracing support. * Add initial forward slash * Actually this is how we add back the initial forward slash
mwfrost · Apr 6, 2018 · 5c894eb · 5c894eb
1 parent 46f5c45
commit 5c894eb
Show file tree

Hide file tree

Showing 3 changed files with 145 additions and 24 deletions.
diff --git a/package.json b/package.json
@@ -16,7 +16,8 @@
     "mime": "^2.2.0",
     "pretty-bytes": "^4.0.2",
     "slash": "^1.0.0",
-    "valid-url": "^1.0.9"
+    "valid-url": "^1.0.9",
+    "xstate": "^3.1.0"
   },
   "devDependencies": {
     "@babel/cli": "^7.0.0-beta.42",

diff --git a/src/create-file-node.js b/src/create-file-node.js
@@ -7,13 +7,6 @@ const prettyBytes = require(`pretty-bytes`)
 const md5File = require(`bluebird`).promisify(require(`md5-file`))
 const crypto = require(`crypto`)
 
-const createId = path => {
-  const slashed = slash(path)
-  return `${slashed} absPath of file`
-}
-
-exports.createId = createId
-
 exports.createFileNode = async (
   pathToFile,
   createNodeId,

diff --git a/src/gatsby-node.js b/src/gatsby-node.js
@@ -1,10 +1,77 @@
 const chokidar = require(`chokidar`)
 const fs = require(`fs`)
+const { Machine } = require(`xstate`)
 
 const { createId, createFileNode } = require(`./create-file-node`)
 
+/**
+ * Create a state machine to manage Chokidar's not-ready/ready states and for
+ * emitting file system events into Gatsby.
+ *
+ * On the latter, this solves the problem where if you call createNode for the
+ * same File node in quick succession, this can leave Gatsby's internal state
+ * in disarray causing queries to fail. The latter state machine tracks when
+ * Gatsby is "processing" a node update or when it's "idle". If updates come in
+ * while Gatsby is processing, we queue them until the system returns to an
+ * "idle" state.
+ */
+const fsMachine = Machine({
+  key: "emitFSEvents",
+  parallel: true,
+  strict: true,
+  states: {
+    CHOKIDAR: {
+      initial: `CHOKIDAR_NOT_READY`,
+      states: {
+        CHOKIDAR_NOT_READY: {
+          on: {
+            CHOKIDAR_READY: "CHOKIDAR_WATCHING",
+            BOOTSTRAP_FINISHED: "CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED",
+          },
+        },
+        CHOKIDAR_WATCHING: {
+          on: {
+            BOOTSTRAP_FINISHED: "CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED",
+            CHOKIDAR_READY: "CHOKIDAR_WATCHING",
+          },
+        },
+        CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED: {
+          on: {
+            CHOKIDAR_READY: "CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED",
+          },
+        },
+      },
+    },
+    PROCESSING: {
+      initial: `BOOTSTRAPPING`,
+      states: {
+        BOOTSTRAPPING: {
+          on: {
+            BOOTSTRAP_FINISHED: "IDLE",
+          },
+        },
+        IDLE: {
+          on: {
+            EMIT_FS_EVENT: `PROCESSING`,
+          },
+        },
+        PROCESSING: {
+          on: {
+            QUERY_QUEUE_DRAINED: `IDLE`,
+            TOUCH_NODE: `IDLE`,
+          },
+        },
+      },
+    },
+  },
+})
+
+let currentState = fsMachine.initialState
+
+const fileQueue = new Map()
+
 exports.sourceNodes = (
-  { actions, getNode, createNodeId, hasNodeChanged, reporter },
+  { actions, getNode, createNodeId, hasNodeChanged, reporter, emitter },
   pluginOptions
 ) => {
   const { createNode, deleteNode } = actions
@@ -21,8 +88,36 @@ Please pick a path to an existing directory.
 See docs here - https://www.gatsbyjs.org/packages/gatsby-source-filesystem/
       `)
   }
+  let fileNodeQueue = new Map()
+
+  // Once bootstrap is finished, we only let one File node update go through
+  // the system at a time.
+  emitter.on(`BOOTSTRAP_FINISHED`, () => {
+    currentState = fsMachine.transition(
+      currentState.value,
+      `BOOTSTRAP_FINISHED`
+    )
+  })
+  emitter.on(`TOUCH_NODE`, () => {
+    // If we create a node which is the same as the previous version, createNode
+    // returns TOUCH_NODE and then nothing else happens so we listen to that
+    // to return the state back to IDLE.
+    currentState = fsMachine.transition(currentState.value, `TOUCH_NODE`)
+  })
 
-  let ready = false
+  emitter.on(`QUERY_QUEUE_DRAINED`, () => {
+    currentState = fsMachine.transition(
+      currentState.value,
+      `QUERY_QUEUE_DRAINED`
+    )
+    // If we have any updates queued, run one of them now.
+    if (fileNodeQueue.size > 0) {
+      const toProcess = fileNodeQueue.get(Array.from(fileNodeQueue.keys())[0])
+      fileNodeQueue.delete(toProcess.id)
+      currentState = fsMachine.transition(currentState.value, `EMIT_FS_EVENT`)
+      createNode(toProcess)
+    }
+  })
 
   const watcher = chokidar.watch(pluginOptions.path, {
     ignored: [
@@ -36,8 +131,21 @@ See docs here - https://www.gatsbyjs.org/packages/gatsby-source-filesystem/
     ],
   })
 
-  const createAndProcessNode = path =>
-    createFileNode(path, createNodeId, pluginOptions).then(createNode)
+  const createAndProcessNode = path => {
+    const fileNodePromise = createFileNode(
+      path,
+      createNodeId,
+      pluginOptions
+    ).then(fileNode => {
+      if (currentState.value.PROCESSING === `PROCESSING`) {
+        fileNodeQueue.set(fileNode.id, fileNode)
+      } else {
+        currentState = fsMachine.transition(currentState.value, `EMIT_FS_EVENT`)
+        createNode(fileNode)
+      }
+    })
+    return fileNodePromise
+  }
 
   // For every path that is reported before the 'ready' event, we throw them
   // into a queue and then flush the queue when 'ready' event arrives.
@@ -50,49 +158,68 @@ See docs here - https://www.gatsbyjs.org/packages/gatsby-source-filesystem/
   }
 
   watcher.on(`add`, path => {
-    if (ready) {
-      reporter.info(`added file at ${path}`)
+    if (currentState.value.CHOKIDAR !== `CHOKIDAR_NOT_READY`) {
+      if (
+        currentState.value.CHOKIDAR === `CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED`
+      ) {
+        reporter.info(`added file at ${path}`)
+      }
       createAndProcessNode(path).catch(err => reporter.error(err))
     } else {
       pathQueue.push(path)
     }
   })
 
   watcher.on(`change`, path => {
-    reporter.info(`changed file at ${path}`)
+    if (
+      currentState.value.CHOKIDAR === `CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED`
+    ) {
+      reporter.info(`changed file at ${path}`)
+    }
     createAndProcessNode(path).catch(err => reporter.error(err))
   })
 
   watcher.on(`unlink`, path => {
-    reporter.info(`file deleted at ${path}`)
-    const node = getNode(createId(path))
+    if (
+      currentState.value.CHOKIDAR === `CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED`
+    ) {
+      reporter.info(`file deleted at ${path}`)
+    }
+    const node = getNode(createNodeId(path))
     // It's possible the file node was never created as sometimes tools will
     // write and then immediately delete temporary files to the file system.
     if (node) {
+      currentState = fsMachine.transition(currentState.value, `EMIT_FS_EVENT`)
       deleteNode(node.id, node)
     }
   })
 
   watcher.on(`addDir`, path => {
-    if (ready) {
-      reporter.info(`added directory at ${path}`)
+    if (currentState.value.CHOKIDAR !== `CHOKIDAR_NOT_READY`) {
+      if (
+        currentState.value.CHOKIDAR === `CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED`
+      ) {
+        reporter.info(`added directory at ${path}`)
+      }
       createAndProcessNode(path).catch(err => reporter.error(err))
     } else {
       pathQueue.push(path)
     }
   })
 
   watcher.on(`unlinkDir`, path => {
-    reporter.info(`directory deleted at ${path}`)
-    const node = getNode(createId(path))
+    if (
+      currentState.value.CHOKIDAR === `CHOKIDAR_WATCHING_BOOTSTRAP_FINISHED`
+    ) {
+      reporter.info(`directory deleted at ${path}`)
+    }
+    const node = getNode(createNodeId(path))
     deleteNode(node.id, node)
   })
 
   return new Promise((resolve, reject) => {
     watcher.on(`ready`, () => {
-      if (ready) return
-
-      ready = true
+      currentState = fsMachine.transition(currentState.value, `CHOKIDAR_READY`)
       flushPathQueue().then(resolve, reject)
     })
   })