From c888c8e1f4328ddd561767bfae2dbb9da6fb9fbc Mon Sep 17 00:00:00 2001 From: Yuliy Pisetsky Date: Fri, 17 May 2024 15:03:17 -0700 Subject: [PATCH 1/5] [gateway] Clarify Reconnect opcode, Document zstd-stream --- docs/topics/Gateway.md | 16 ++++++++++++---- docs/topics/Gateway_Events.md | 2 +- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/topics/Gateway.md b/docs/topics/Gateway.md index fa61d5fe0e..e380edfb45 100644 --- a/docs/topics/Gateway.md +++ b/docs/topics/Gateway.md @@ -80,7 +80,7 @@ At a high-level, Gateway connections consist of the following cycle: - Discord may send the app a [Heartbeat (opcode `1`)](#DOCS_TOPICS_GATEWAY_EVENTS/heartbeat) event, in which case the app should send a Heartbeat event immediately. 4. App sends an [Identify (opcode `2`)](#DOCS_TOPICS_GATEWAY_EVENTS/identify) event to perform the initial handshake with the Gateway. **Read the section on [Identifying](#DOCS_TOPICS_GATEWAY/identifying)** 5. Discord sends the app a [Ready (opcode `0`)](#DOCS_TOPICS_GATEWAY_EVENTS/ready) event which indicates the handshake was successful and the connection is established. The Ready event contains a `resume_gateway_url` that the app should keep track of to determine the WebSocket URL an app should use to Resume. **Read the section on [the Ready event](#DOCS_TOPICS_GATEWAY/ready-event)** -6. The connection may be dropped for a variety of reasons. Whether the app can [Resume](#DOCS_TOPICS_GATEWAY/resuming) the connection or whether it must re-identify is determined by a variety of factors like the [opcode](#DOCS_TOPICS_OPCODES_AND_STATUS_CODES/gateway-gateway-opcodes) and [close code](#DOCS_TOPICS_OPCODES_AND_STATUS_CODES/gateway-gateway-close-event-codes) that it receives. **Read the section on [Disconnecting](#DOCS_TOPICS_GATEWAY/disconnecting)** +6. The connection may be dropped for a variety of reasons at any time. Whether the app can [Resume](#DOCS_TOPICS_GATEWAY/resuming) the connection or whether it must re-identify is determined by a variety of factors like the [opcode](#DOCS_TOPICS_OPCODES_AND_STATUS_CODES/gateway-gateway-opcodes) and [close code](#DOCS_TOPICS_OPCODES_AND_STATUS_CODES/gateway-gateway-close-event-codes) that it receives. **Read the section on [Disconnecting](#DOCS_TOPICS_GATEWAY/disconnecting)** 7. If an app **can** resume/reconnect, it should open a new connection using `resume_gateway_url` with the same version and encoding, then send a [Resume (opcode `6`)](#DOCS_TOPICS_GATEWAY_EVENTS/resume) event. If an app **cannot** resume/reconnect, it should open a new connection using the cached URL from step #1, then repeat the whole Gateway cycle. *Yipee!* **Read the section on [Resuming](#DOCS_TOPICS_GATEWAY/resuming)** ### Connecting @@ -478,7 +478,7 @@ Apps also have a limit for [concurrent](#DOCS_TOPICS_GATEWAY/session-start-limit When [establishing a connection](#DOCS_TOPICS_GATEWAY/connecting) to the Gateway, apps can use the `encoding` parameter to choose whether to communicate with Discord using a plain-text JSON or binary [ETF](https://erlang.org/doc/apps/erts/erl_ext_dist.html) encoding. You can pick whichever encoding type you're more comfortable with, but both have their own quirks. If you aren't sure which encoding to use, JSON is generally recommended. -Apps can also optionally enable compression to receive zlib-compressed packets. [Payload compression](#DOCS_TOPICS_GATEWAY/payload-compression) can only be enabled when using a JSON encoding, but [transport compression](#DOCS_TOPICS_GATEWAY/transport-compression) can be used regardless of encoding type. +Apps can also optionally enable compression to receive zlib-compressed or zstd-compressed packets. [Payload compression](#DOCS_TOPICS_GATEWAY/payload-compression) can only be enabled when using a JSON encoding, but [transport compression](#DOCS_TOPICS_GATEWAY/transport-compression) can be used regardless of encoding type. ### Using JSON Encoding @@ -509,9 +509,11 @@ See [erlpack](https://github.com/discord/erlpack) for an ETF implementation exam ### Transport Compression -Transport compression enables optional compression for all packets when Discord is sending events over the connection. The only currently-available transport compression option is `zlib-stream`. +Transport compression enables optional compression for all packets when Discord is sending events over the connection. The only currently-available transport compression options are `zlib-stream` and `zstd-stream`. -When transport compression is enabled, your app needs to process received data through a single Gateway connection using a shared zlib context. However, each Gateway connection should use its own unique zlib context. +#### zlib-stream + +When zlib transport compression is enabled, your app needs to process received data through a single Gateway connection using a shared zlib context. However, each Gateway connection should use its own unique zlib context. When processing transport-compressed data, you should push received data to a buffer until you receive the 4-byte `Z_SYNC_FLUSH` suffix (`00 00 ff ff`). After you receive the `Z_SYNC_FLUSH` suffix, you can then decompress the buffer. @@ -544,6 +546,12 @@ def on_websocket_message(msg): # depending on your `encoding` param ``` +#### zstd-stream + +When zstd-stream transport compression is enabled, all data needs to be processed through a zstd decompression context that stays alive for the lifetime of the gateway connection. + +When processing data, each websocket message corresponds to a single gateway message, but does not end a zstd frame. You will need to repeatedly call ZSTD_decompressStream until all data in the frame has been processed (ZSTD_decompressStream will not necessarily return 0, though). Take a look at this [Erlang](https://github.com/silviucpp/ezstd/blob/f3f33b2f6b917f7e8aaa2b4d71338620537df81b/src/ezstd.erl#L151-L169) + [C++](https://github.com/silviucpp/ezstd/blob/f3f33b2f6b917f7e8aaa2b4d71338620537df81b/c_src/ezstd_nif.cc#L520-L568) implementation for inspiration. + ## Tracking State Most of a client's state is provided during the initial [Ready](#DOCS_TOPICS_GATEWAY/ready-event) event and in the [Guild Create](#DOCS_TOPICS_GATEWAY_EVENTS/guild-create) events that follow. diff --git a/docs/topics/Gateway_Events.md b/docs/topics/Gateway_Events.md index 2baa707eb8..1d5e347061 100644 --- a/docs/topics/Gateway_Events.md +++ b/docs/topics/Gateway_Events.md @@ -387,7 +387,7 @@ The resumed event is dispatched when a client has sent a [resume payload](#DOCS_ #### Reconnect -The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This event usually occurs during deploys to migrate sessions gracefully off old hosts. +The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This can occur at any point in the gateway connection lifecycle. A few seconds after this event is dispatched, the connection may be closed by the server. This may even happen before/in place of receiving a [Hello](#DOCTS_TOPICS_GATEWAY_EVENTS/hello). ###### Example Gateway Reconnect From 26516ba24584e97a375bd1531bffba2d5b1bf4e6 Mon Sep 17 00:00:00 2001 From: Yuliy Pisetsky Date: Fri, 17 May 2024 16:14:01 -0700 Subject: [PATCH 2/5] fix typo --- docs/topics/Gateway_Events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/Gateway_Events.md b/docs/topics/Gateway_Events.md index 1d5e347061..70983a3c55 100644 --- a/docs/topics/Gateway_Events.md +++ b/docs/topics/Gateway_Events.md @@ -387,7 +387,7 @@ The resumed event is dispatched when a client has sent a [resume payload](#DOCS_ #### Reconnect -The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This can occur at any point in the gateway connection lifecycle. A few seconds after this event is dispatched, the connection may be closed by the server. This may even happen before/in place of receiving a [Hello](#DOCTS_TOPICS_GATEWAY_EVENTS/hello). +The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This can occur at any point in the gateway connection lifecycle. A few seconds after this event is dispatched, the connection may be closed by the server. This may even happen before/in place of receiving a [Hello](#DOCS_TOPICS_GATEWAY_EVENTS/hello). ###### Example Gateway Reconnect From 2d1cd792fe80de10998626466709b13c97180826 Mon Sep 17 00:00:00 2001 From: Yuliy Pisetsky Date: Fri, 17 May 2024 16:27:50 -0700 Subject: [PATCH 3/5] also add zstd-stream to query string docs --- docs/topics/Gateway.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/Gateway.md b/docs/topics/Gateway.md index e380edfb45..4888ff0a31 100644 --- a/docs/topics/Gateway.md +++ b/docs/topics/Gateway.md @@ -103,7 +103,7 @@ When connecting to the URL, it's a good idea to explicitly pass the API version |-----------|---------|-----------------------------------------------------------------------------------------------------|------------------------------------------------------------| | v | integer | [API Version](#DOCS_REFERENCE/api-versioning) to use | [API version](#DOCS_REFERENCE/api-versioning-api-versions) | | encoding | string | The [encoding](#DOCS_TOPICS_GATEWAY/encoding-and-compression) of received gateway packets | `json` or `etf` | -| compress? | string | The optional [transport compression](#DOCS_TOPICS_GATEWAY/transport-compression) of gateway packets | `zlib-stream` | +| compress? | string | The optional [transport compression](#DOCS_TOPICS_GATEWAY/transport-compression) of gateway packets | `zlib-stream` or `zstd-stream` | #### Hello Event From 5f19e4510f33f26f1f96e8f2f1361476088407fb Mon Sep 17 00:00:00 2001 From: Yuliy Pisetsky Date: Fri, 17 May 2024 17:35:34 -0700 Subject: [PATCH 4/5] Update docs/topics/Gateway.md Co-authored-by: advaith --- docs/topics/Gateway.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/Gateway.md b/docs/topics/Gateway.md index 4888ff0a31..c2d0c2d777 100644 --- a/docs/topics/Gateway.md +++ b/docs/topics/Gateway.md @@ -509,7 +509,7 @@ See [erlpack](https://github.com/discord/erlpack) for an ETF implementation exam ### Transport Compression -Transport compression enables optional compression for all packets when Discord is sending events over the connection. The only currently-available transport compression options are `zlib-stream` and `zstd-stream`. +Transport compression enables optional compression for all packets when Discord is sending events over the connection. The currently-available transport compression options are `zlib-stream` and `zstd-stream`. #### zlib-stream From 69df12ccd9aad0e6c832c0cf11ebc602b985a481 Mon Sep 17 00:00:00 2001 From: Yuliy Pisetsky Date: Thu, 23 May 2024 09:18:34 -0700 Subject: [PATCH 5/5] Update docs/topics/Gateway_Events.md Co-authored-by: val.le <81811276+valdotle@users.noreply.github.com> --- docs/topics/Gateway_Events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/Gateway_Events.md b/docs/topics/Gateway_Events.md index 70983a3c55..43d6997719 100644 --- a/docs/topics/Gateway_Events.md +++ b/docs/topics/Gateway_Events.md @@ -387,7 +387,7 @@ The resumed event is dispatched when a client has sent a [resume payload](#DOCS_ #### Reconnect -The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This can occur at any point in the gateway connection lifecycle. A few seconds after this event is dispatched, the connection may be closed by the server. This may even happen before/in place of receiving a [Hello](#DOCS_TOPICS_GATEWAY_EVENTS/hello). +The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This can occur at any point in the gateway connection lifecycle, even before/in place of receiving a [Hello](#DOCS_TOPICS_GATEWAY_EVENTS/hello) event. A few seconds after the reconnect event is dispatched, the connection may be closed by the server. ###### Example Gateway Reconnect