[RFC] Persisted Document support #109

alangenfeld · 2016-07-20T22:33:20Z

This adds support to express-graphql for GraphQL persisted documents. This is a feature we have discussed in ad-hoc ways since open sourcing GraphQL, and are now attempting to demonstrate more concretely. Persisted documents are foundational to how we use GraphQL at Facebook, especially for our mobile clients. Additional background context is being added to the README.

The goal in adding this support to express-graphql is to provide a clear example of how this idea can be implemented which should allow it to be translated to any other GraphQL server middleware or implementation.

Notes:

persistValidatedDocument and providing the /persist endpoint is still a nascent idea, that needs the most scrutiny. Then once its settled, tests.

alangenfeld · 2016-07-20T22:35:05Z

I am not the greatest JS engineer, so basic/style feedback is expected and welcome.

coveralls · 2016-07-20T22:45:26Z

Coverage decreased (-16.8%) to 83.178% when pulling 370cdad on alangenfeld:persisted-document into 6dcf37a on graphql:master.

kassens · 2016-07-21T16:30:53Z

src/index.js

+  request: Request,
+  response: Response
+): Promise<OptionsData> {
+  return new Promise(resolve => {


No need for new Promise:

return Promise.resolve( typeof options === 'function' ? options(request, response) : options ).then

tried that, but doing so prevents us from catching exceptions thrown from options when it is a function at [1]

nodkz · 2016-08-20T04:52:21Z

@alangenfeld @wincent @josephsavona why persisted document?

Document can't have variables for fetching. According to provided variables, we may get the absolutely different responses for same document (query) id.

I think better naming is persisted query. Word document brings to much ambiguity.

dschafer · 2016-08-20T05:17:21Z

Yeah, this is a tricky nomenclature issue. The parameter is params.query, but params.query doesn't have to contain a GraphQL Query; it has to contain a GraphQL Document. So I could see either name working here.

I'm not sure Document is ambiguous in this context, since it refers to http://facebook.github.io/graphql/#sec-Language.Query-Document (or to the Document flowtype in graphql-js), but it doesn't match the existing parameter naming.

nodkz · 2016-08-20T05:59:15Z

QueryDocument will be a nice name, but not just document. Anyway for me, PersistedQuery is the best naming.

@dschafer please consider better name for flowtype parameter in graphql-js. Document is not good.

lacker · 2016-09-16T06:39:57Z

README.md

+    returns an id that can be used to load it later. This is used in conjunction
+    with `loadPersistedDocument`. Providing this function will enable persisting
+    documents via the `document` parameter at the `/persist` subpath. It is
+    recommended that this option only be enabled in development deployments.


It's not clear to me when you would want to use persistValidatedDocument in development. You basically have to be storing the persisted documents either in some database, or in your source code, right? And either way, hitting a persistValidatedDocument REST API on a development backend server is not going to be the easiest way to get these persisted documents into the place they need to go. In your development environment you can just write them to the db or put them in your source code directly.

If you want to build tooling for client development, you might want to include persisting all required client queries into your build process and have an API for this, making this interface HTTP is pretty obvious.

Persisting them to a database has the nice benefit (especially for larger organizations) that you don't need a new server deployment after adding (updating) a new query. You could even imagine that Github's public GraphQL API allowed clients to persist queries.

You would have to be running your backend server during your build process, though - isn't that somewhat unusual? Rather than writing some backend code to persist the document, and making your build process hit an HTTP API on your running backend server, your build process could just execute the code to persist the document directly. I guess overall it seems like this is something people could implement themselves just as easily as if we added a persistValidatedDocument API.

I don't think it's unusual at all - in fact it seems the opposite to me. When building my iOS app or Android app or web app, I'd be really surprised if in that context I couldn't access my GraphQL API because it was also being built. I would expect a GraphQL API to be decoupled from the apps that are using it such that the build and deploy processes are also separate. Typically the build process for an iOS, Android or web app does not have access to a database, which is why exposing an HTTP API is beneficial.

lacker · 2016-09-16T06:43:52Z

README.md

@@ -53,6 +53,16 @@ The `graphqlHTTP` function accepts the following options:
  * **`graphiql`**: If `true`, may present [GraphiQL][] when loaded directly
    from a browser (a useful tool for debugging and exploration).

+  * **`loadPersistedDocument`**: A function that takes an input id and returns a
+    valid Document. If provided, this will allow your GraphQL endpoint to execute
+    a document specified via `documentID`.


These documents typically each have a unique name, right? Why not use the name as the identifier instead of a separate documentID? I think that is how Airbnb is setting up their graphql backend.

While the document might have a unique name like HomePageQuery, this is not enough, as the query likely changes over time as new client versions are released. It is important that old clients still get the old version of HomePageQuery.

lacker · 2016-09-16T06:44:13Z

README.md

@@ -156,6 +166,22 @@ new GraphQLObjectType({
 });
 ```

+## Persisted Documents
+
+Putting control of the query in to clients hands is one of the primary benefits


in to -> into
clients -> clients'

josephsavona · 2016-09-16T15:31:36Z

@lacker the problem with using the document name as the identifier is that the document changes over time, the id serves as a hash for a rev of a particular document.

alangenfeld · 2016-09-21T20:27:26Z

Apologies for the long delay, its been a hectic few months in GraphQL land at FB. Thanks everybody for the feedback so far. I think the questions that are still open are:

naming: persisted document vs. persisted query
whether supporting /graphql/persist?document= if you provide persistValidatedDocument makes sense

lacker · 2016-09-27T23:30:17Z

My suggestions + rationale for the open questions:

"persisted query" because in the graphql community more people refer to them as "queries" than as "documents"
no support for /graphql/persist?document=. It doesn't seem like this provides enough benefit over people implementing their own endpoint.

helfer · 2016-09-27T23:43:13Z

In apollo-server we called them "stored operations", since that encompasses queries, mutations and subscriptions. I know that technically they can also contain fragments, which makes them documents, but at least it's a less confusing name than "queries". Using "queries" may lead people to believe that mutations can't be persisted.

leebyron · 2016-11-03T02:33:51Z

Let's revive this and get it shored up and ready to land.

Some thoughts:

"Document" is absolutely the right name. We've been bad about clarity around names and in fact should consider renaming some of the fields accepted in this library to help with that. A Document is a whole GraphQL file or request. An operation is one query/mutation (a document may contain many in addition to fragments). A query is a kind of operation. Since we're typically talking about documents that have one operation that is a query, we say "GraphQL query" - but I think it's important to get the naming right for this which should allow persisting any kind of document.

Perhaps we should rearrange the API such that you provide both or neither functions. Right now we could assert one or the other, but I think the API could help with that. Maybe:

persistedDocuments: {
  save: document => id 
  load: id => document
}

I think we're missing an operation as well, which is getting the actual document text for a given id. Tools like GraphiQL would want to use this to populate the left pane. Maybe we should expose paths savepersisted and loadpersisted to represent both?

One last thought - should we be persisting the actual string text rather than the AST itself? Or perhaps allowing either with a typeof === 'string' check? I'm thinking if your document contains comments or preferred formatting, that a round trip through persistence should retain those.

josh · 2016-11-03T17:44:08Z

should we be persisting the actual string text rather than the AST itself?

I've been curious about the same thing for @github. Right now we're thinking of making our persisted IDs a SHA256 of the document text. Normalizing the document AST would let us store slightly different text documents under the same hashed ID. Seems like a good idea to me. Is there even a mechanize that would allow clients to retrieve the original document text? If they can only persist and execute operations, any comments wouldn't make a difference.

leebyron · 2016-11-03T19:10:41Z

I think it would be valuable to retrieve the original document text for use with tools like GraphiQL. It would be immensely helpful for debugging to be able to take one of these persisted IDs and check what the server is actually running for it.

leebyron · 2016-11-03T19:12:32Z

Also, if you'd prefer to have comments or formatting be irrelevant to the hash of a query, that's certainly something you could have a client decide by doing a hash(print(parse(text))) to normalize first.

JeffRMoore · 2016-11-08T15:48:25Z

Is there some value here to borrowing the prepared statement nomenclature from SQL? Many programmers are already familiar the concept of working with a statement in two phases, one phase to setup the statement and the second phase to execute it with parameters.

The methods in this example might become prepareDocument and loadPreparedDocument?

I wonder if "persist" terminology might require slightly more cognition to disambiguate. It is the communication pattern with the server that is being persisted, not necessarily the data being exchanged.

intellix · 2016-11-13T13:22:11Z

src/index.js

+    resolveOptionsData(options, request, response),
+    parseBody(request)
+  ]).then(initData => {
+    const [ optionsData, bodyData ] = initData;


travis.yml contains Node 0.12 with all tests currently passing. I believe destructuring isn't supported there.

leebyron · 2017-01-24T21:38:59Z

I'll take over from here :)

alangenfeld · 2017-01-24T22:12:46Z

Thanks @leebyron! Apologies for not being able to see this one through.

raderio · 2017-11-01T14:52:34Z

Is someone using approach: on dev environment support mix(persisted queries & arbitrary queries) and on prod environment support only persisted queries?

taion · 2017-12-15T17:05:15Z

What's the status on this PR?

j0k3r · 2018-03-26T05:41:34Z

Any news?

alloy · 2018-04-09T13:32:13Z

@leebyron @kassens We’re now starting to use persisted queries at Artsy and so I’d like to contribute some time to take this PR over the finish line. Could you summarise the work left to do from your perspective?

leebyron

Sorry for letting this sit. I'd love to have your help on this, @alloy.

From looking over the PR it seems that really this just needs a few things, repairing conflicts and cleaning things up a bit would be helpful!

I also think persistValidatedDocument should provide both the AST and the original query string, probably as (text, ast) => id so that the unedited text can be saved and a call to print be the user is unnecessary for most storage cases.

Similarly, It would be nice if loadPersistedDocument accepted a return type of text | ast and used typeof === 'string' to disambiguate

leebyron · 2018-04-09T16:36:29Z

src/__tests__/http-test.js

+        let response = await request(app)
+          .post(persistString({
+            document: '{test}'
+          }));


Need a test that posting data works instead of just formatting a query string

leebyron · 2018-04-09T16:38:48Z

src/index.js

        if (showGraphiQL) {
          return null;
        }
-        throw httpError(400, 'Must provide query string.');
+        throw httpError(400, 'Must provide query string or document ID.');


Should probably only say document ID should be provided if loadPersistedDocument is available

leebyron · 2018-04-09T16:39:49Z

src/index.js

-        // Return 400: Bad Request if any validation errors exist.
-        response.statusCode = 400;
-        return { errors: validationErrors };
+      if (query) {


These 3 if statements are a bit confusing in sequence, I would expect:

if (documentID) { } else if (query) { } else {

leebyron · 2018-04-09T16:40:36Z

src/index.js

@@ -278,7 +285,7 @@ function getGraphQLParams(urlData: Object, bodyData: Object): GraphQLParams {
  // Name of GraphQL operation to execute.
  const operationName = urlData.operationName || bodyData.operationName;

-  return { query, variables, operationName };
+  return { query, variables, operationName, documentID};


formatting (note: we should enable prettier on this repo)

alloy · 2018-04-09T22:27:30Z

@leebyron Looks straightforward enough, thanks! 👍 I’ll start work on that this week.

facebook-github-bot · 2018-07-25T20:59:42Z

Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours has expired.

Before we can review or merge your code, we need you to email cla@fb.com with your details so we can update your status.

sibelius · 2019-01-02T17:46:47Z

let's make this happen in 2019?

facebook-github-bot · 2019-01-02T18:05:27Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

alloy · 2019-01-02T18:38:53Z

I made many attempts at updating this PR, but the lack of context and it being so out of date made it hard :/ It’s definitely still on my TODO list, so yeah maybe this will be the year!

sibelius · 2019-01-05T10:56:47Z

this could be achieve in a middleware based way, so we don't need to make express-graphql more complex

like this https://gist.github.com/robrichard/d5a6ae7ecb3c9efc7fad7b3236d255e7 solution by @robrichard

using the middleware approach makes easy to work with express-graphql, koa-graphql, apollo-server and any other middleware based server

josephsavona · 2019-01-05T16:08:56Z

@sibelius that seems like a great approach

Janpot · 2019-01-05T19:33:01Z

But it should be standardized, no? To guarantee interoperability between the different client and server libraries. Middleware or not is just an implementation detail, right?

alloy · 2019-01-07T17:09:19Z

Yeah agreed that that could be a good approach 👍 I do think we may still need to add some change here (eg to be able to use already parsed and validated documents), but those should be easy to do from scratch.

sibelius · 2019-01-07T17:13:11Z

we could have another package to handle all this persisted queries validation and logic

ghost added the CLA Signed label Jul 20, 2016

kassens reviewed Jul 21, 2016
View reviewed changes

stubailo mentioned this pull request Jul 28, 2016

Proposal: Relay-like containers apollographql/react-apollo#122

Closed

Poincare mentioned this pull request Jul 28, 2016

Persisted queries apollographql/apollo-client#478

Closed

antmdvs mentioned this pull request Aug 18, 2016

[Feature request] Returning fieldASTs to be able to cache them for performance imporvements graphql/graphql-js#158

Closed

lacker reviewed Sep 16, 2016

View reviewed changes

initial support for persisted document

772a1cb

alangenfeld force-pushed the persisted-document branch from 370cdad to 772a1cb Compare September 21, 2016 20:06

fix readme typos

fe9afd9

intellix reviewed Nov 13, 2016

View reviewed changes

robzhu mentioned this pull request Nov 30, 2016

Ideas on how to optimize GraphQL execution (and validation) ? graphql/graphql-js#607

Closed

rricard mentioned this pull request Dec 22, 2016

Umbrella Issue: Server side support for Persisted Queries apollographql/apollo-server#253

Closed

6 tasks

wincent mentioned this pull request Jan 30, 2017

When will we have Persisted Query support in js version of Relay? facebook/relay#1455

Closed

jnwng mentioned this pull request Jan 31, 2017

Babel plugin for graphql-tag apollographql/graphql-tag#31

Closed

martijnwalraven mentioned this pull request Jun 24, 2017

Generate Swift operation IDs and operation <--> operation ID mapping file apollographql/apollo-tooling#147

Merged

3 tasks

johntran mentioned this pull request Aug 2, 2017

[modern] Persisted GraphQL Queries in Modern? facebook/relay#2008

Open

wincent mentioned this pull request Aug 31, 2017

Query complexity analysis, protection against malicious queries graphql/graphql-js#978

Closed

alloy mentioned this pull request Apr 9, 2018

[persisted queries] Lookup queries in JSON as initial implementation. artsy/metaphysics#995

Merged

leebyron reviewed Apr 9, 2018

View reviewed changes

Base automatically changed from master to main February 10, 2021 15:09

alangenfeld closed this Jul 22, 2021

[RFC] Persisted Document support #109

[RFC] Persisted Document support #109

Conversation

alangenfeld commented Jul 20, 2016 • edited Loading

alangenfeld commented Jul 20, 2016

coveralls commented Jul 20, 2016

kassens Jul 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodkz commented Aug 20, 2016

dschafer commented Aug 20, 2016

nodkz commented Aug 20, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephsavona commented Sep 16, 2016

alangenfeld commented Sep 21, 2016

lacker commented Sep 27, 2016

helfer commented Sep 27, 2016

leebyron commented Nov 3, 2016

josh commented Nov 3, 2016

leebyron commented Nov 3, 2016

leebyron commented Nov 3, 2016 • edited Loading

JeffRMoore commented Nov 8, 2016

Choose a reason for hiding this comment

leebyron commented Jan 24, 2017

alangenfeld commented Jan 24, 2017

raderio commented Nov 1, 2017

taion commented Dec 15, 2017

j0k3r commented Mar 26, 2018

alloy commented Apr 9, 2018

leebyron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alloy commented Apr 9, 2018

facebook-github-bot commented Jul 25, 2018

sibelius commented Jan 2, 2019

facebook-github-bot commented Jan 2, 2019

alloy commented Jan 2, 2019

sibelius commented Jan 5, 2019

josephsavona commented Jan 5, 2019

Janpot commented Jan 5, 2019 • edited Loading

alloy commented Jan 7, 2019 • edited Loading

sibelius commented Jan 7, 2019

alangenfeld commented Jul 20, 2016 •

edited

Loading

kassens Jul 21, 2016 •

edited

Loading

nodkz commented Aug 20, 2016 •

edited

Loading

leebyron commented Nov 3, 2016 •

edited

Loading

Janpot commented Jan 5, 2019 •

edited

Loading

alloy commented Jan 7, 2019 •

edited

Loading