[json-loader] develop - reading results from file improvments #4850
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y if we don't have it stored yet (we didn't run this query, but results are cached)
ghost
assigned 
|
LGTM |
|
Merging this in — then going to test the json-loader branch before merging that into v2 🎉 💃 |
|
LGTM too! Edit: merged while I was reading 🎉 |
KyleAMathews
pushed a commit
that referenced
this pull request
Apr 6, 2018
…ion (aka Ludicrous Mode) (#4555) * Create placeholder JSON store * Rename * Websocket placeholder * Push query results JSON over websockets * More descriptive variable name * Fix queries being overwritten * Remove eslint-disable flag * Remove junk * test require error fix for windows * dont require json data in sync-require * dont add layout data to json array multiple times * initial async loading * revert saving json directly to public for now * updated production-app to sync with prop name change in ComponentRenderer * we load json data via json-loader component in develop and not handling it with webpack import/require * hashes for json files * fix preloading, use xhr instead of fetch - for some reason can't force fetch to not create additional request, with any `cache` or `mode` configuration * dont use full paths in dataPath - remove static/d/ path and .json ext - results in smaller app bundle especially with large ammount of pages * Enable cached query results to be loaded * Don't dump all query results out to the client Instead only push results out if the data is for a path that's currently being viewed in a client. * fix preload link to json data * remove not used function * remove more not used code * Update to latest webpack/mini-css-extract-plugin * don't write new (a)sync-requires.js if components didn't change (#4759) * create just one websocket client (#4763) * Filter out duplicate query jobs and create secondary queue for jobs if path already has query in flight * [json-loader] Don't emit new file node until previous is finished processing (#4785) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * [json-loader] Only log file events if we're past bootstrap (#4826) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * Only log file events if we're past bootstrap * [json-loader] dont recompile on data change - part 2 (#4837) * prevent adding duplicate redirects * don't write new `redirects.json` if redirects didn't change prevents webpack recompilation on data change * [json-loader] develop - reading results from file improvments (#4850) * dont emit results for layouts * [develop] store query results in memory, read json data from file only if we don't have it stored yet (we didn't run this query, but results are cached) * Add query prioritization based on what page(s) user(s) are on Query running is sadly not very ludicrous right now on gatsbyjs.org — not sure why — each markdown file change causes ~20 queries to run but even with prioritizing the active page's query, it's still ~2 seconds before the page updates. This sort of thing will be much easier to debug with tracing support. * Add initial forward slash * Actually this is how we add back the initial forward slash
mwfrost
pushed a commit
to mwfrost/gatsby
that referenced
this pull request
Apr 20, 2023
…ion (aka Ludicrous Mode) (gatsbyjs#4555) * Create placeholder JSON store * Rename * Websocket placeholder * Push query results JSON over websockets * More descriptive variable name * Fix queries being overwritten * Remove eslint-disable flag * Remove junk * test require error fix for windows * dont require json data in sync-require * dont add layout data to json array multiple times * initial async loading * revert saving json directly to public for now * updated production-app to sync with prop name change in ComponentRenderer * we load json data via json-loader component in develop and not handling it with webpack import/require * hashes for json files * fix preloading, use xhr instead of fetch - for some reason can't force fetch to not create additional request, with any `cache` or `mode` configuration * dont use full paths in dataPath - remove static/d/ path and .json ext - results in smaller app bundle especially with large ammount of pages * Enable cached query results to be loaded * Don't dump all query results out to the client Instead only push results out if the data is for a path that's currently being viewed in a client. * fix preload link to json data * remove not used function * remove more not used code * Update to latest webpack/mini-css-extract-plugin * don't write new (a)sync-requires.js if components didn't change (gatsbyjs#4759) * create just one websocket client (gatsbyjs#4763) * Filter out duplicate query jobs and create secondary queue for jobs if path already has query in flight * [json-loader] Don't emit new file node until previous is finished processing (gatsbyjs#4785) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * [json-loader] Only log file events if we're past bootstrap (gatsbyjs#4826) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * Only log file events if we're past bootstrap * [json-loader] dont recompile on data change - part 2 (gatsbyjs#4837) * prevent adding duplicate redirects * don't write new `redirects.json` if redirects didn't change prevents webpack recompilation on data change * [json-loader] develop - reading results from file improvments (gatsbyjs#4850) * dont emit results for layouts * [develop] store query results in memory, read json data from file only if we don't have it stored yet (we didn't run this query, but results are cached) * Add query prioritization based on what page(s) user(s) are on Query running is sadly not very ludicrous right now on gatsbyjs.org — not sure why — each markdown file change causes ~20 queries to run but even with prioritizing the active page's query, it's still ~2 seconds before the page updates. This sort of thing will be much easier to debug with tracing support. * Add initial forward slash * Actually this is how we add back the initial forward slash
This was referenced Dec 1, 2023
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
this remove excessive file reads: