Merge pull request #497 from Kraigie/configurable-shards

Configurable shards

guides/intro/intro.md

@@ -35,55 +35,73 @@ config :nostrum,
 
 ## Configuration options
 
-Apart from the fields mentioned above, the following fields are also supported:
-
- - `num_shards` - A fixed number of shards to run, or `:auto` to have Nostrum
-   determine it automatically. Defaults to `:auto`.
- - `ffmpeg` - Specifies the path to the `ffmpeg` executable for playing audio.
-   Defaults to `"ffmpeg"`.
- - `youtubedl` - Specifies the path to the `youtube-dl` executable for playing
-   audio with youtube-dl support. Defaults to `"youtube-dl"`.
- - `streamlink` - Specifies the path to the `streamlink` executable for playing
-   livestream audio with streamlink support. Defaults to `"streamlink"`.
- - `gateway_intents` - This field takes a list of atoms representing gateway
-   intents for Nostrum to subscribe to from the Discord API. More information
-   can be found in the [gateway intents](gateway-intents.html) documentation
-   page.
- - `audio_timeout` - Milliseconds that input must begin generating audio by
-   upon invoking `play`. More information about this option can be found in the
-   [voice](voice.html) documentation page. Defaults to `20_000` (20s).
- - `audio_frames_per_burst` - Number of opus frames to send at a time while
-   playing audio. More information about this option can be found in the
-   [voice](voice.html) documentation page. Defaults to `10`.
- - `voice_auto_connect` - This will determine if Nostrum automatically connects
-   to voice websockets gateways upon joining voice channels. If set to `false`
-   but you still wish to connect to the voice gateway, you can do so manually
-   by calling `Nostrum.Voice.connect_to_gateway/1` after joining a voice
-   channel. Defaults to `true`.
- - `dev` - This is added to enable Nostrum to be run completely stand alone for
-   development purposes. `true` will cause Nostrum to spawn its own event
-   consumers. If you have the dev flag set to true while running Nostrum
-   alongside your application some of your events will be consumed. Defaults to
-   `false`.
- - `log_full_events` - This will log the full payload received over the
-   websocket. This is included primarily for debugging and testing purposes.
-   Defaults to `false`.
- - `log_dispatch_events` - This will log dispatch events as they are received
-   from the gateway. This is included primarily for debugging and testing
-   purposes. Defaults to `false`. 
- - `request_guild_members` - This will perform member chunking to retrieve a
-   complete list of members for all guilds. This will increase start up time
-   and memory usage by quite a bit. Defaults to `false`.
- - `fullsweep_after_default` - Sets the `fullsweep_after` flag for processes
-   that can have irregularly high memory usage due to Discord payloads. This
-   options will dramatically reduce the amount of memory used by some processes
-   at the cost of increased CPU usage. This is useful if you're running your
-   application under a memory constrained environment. This comes at the cost
-   of increased CPU usage. By default, this option will only affect some
-   processes. You can set this flag for *all* processes using environment
-   variables or by [setting the system flag
-   yourself](http://erlang.org/doc/man/erlang.html#system_flag-2). Defaults to
-   whatever your system recommends, which is probably `65535`.
+Apart from the `token` field mentioned above, the following fields are also supported:
+
+- `num_shards` - the amount of shards to run. Can be one of the following:
+  - `:auto`: use the suggested amount of shards as provided by Discord.
+  - *`num`*: a number of shards to run. nostrum will warn if this is not the
+    recommended amount.
+  - `{lowest, highest, total}`: start shards `lowest` to `highest`. `total`
+    should contain the total amount of shards that your bot is expected to have.
+    Useful for splitting a single bot across multiple servers, but see also [the
+    multi-node documentation](../advanced/multi_node.md).
+- `gateway_intents` - a list of atoms representing gateway intents for Nostrum
+  to subscribe to from the Discord API. More information can be found in the
+  [gateway intents](./gateway_intents.md) documentation page.
+- `request_guild_members` - perform member chunking to retrieve a complete list
+  of members for all guilds at startup. Depending on your [cache
+  backend](../advanced/pluggable_caching.md), this may increase startup time
+  and, memory usage by quite a bit. Defaults to `false`.
+
+
+### Voice-specific
+
+- `ffmpeg` - Specifies the path to the `ffmpeg` executable for playing audio.
+  Defaults to `"ffmpeg"`.
+- `youtubedl` - Specifies the path to the `youtube-dl` executable for playing
+  audio with youtube-dl support. Defaults to `"youtube-dl"`.
+- `streamlink` - Specifies the path to the `streamlink` executable for playing
+  livestream audio with streamlink support. Defaults to `"streamlink"`.
+- `audio_timeout` - Milliseconds that input must begin generating audio by
+  upon invoking `play`. More information about this option can be found in the
+  [voice](./voice.html) documentation page. Defaults to `20_000` (20s).
+- `audio_frames_per_burst` - Number of opus frames to send at a time while
+  playing audio. More information about this option can be found in the
+  [voice](./voice.html) documentation page. Defaults to `10`.
+- `voice_auto_connect` - This will determine if Nostrum automatically connects
+  to voice websockets gateways upon joining voice channels. If set to `false`
+  but you still wish to connect to the voice gateway, you can do so manually
+  by calling `Nostrum.Voice.connect_to_gateway/1` after joining a voice
+  channel. Defaults to `true`.
+
+
+### Development & debugging
+
+- `log_full_events` - This will log the full payload received over the
+  websocket. Defaults to `false`.
+- `log_dispatch_events` - This will log dispatch events as they are received
+  from the gateway. Defaults to `false`.
+- `fullsweep_after_default` - Sets the `fullsweep_after` flag for processes
+  that can have irregularly high memory usage due to Discord payloads. This
+  options will dramatically reduce the amount of memory used by some processes
+  at the cost of increased CPU usage. This is useful if you're running your
+  application under a memory constrained environment. This comes at the cost
+  of increased CPU usage. By default, this option will only affect some
+  processes. You can set this flag for *all* processes using environment
+  variables or by [setting the system flag
+  yourself](http://erlang.org/doc/man/erlang.html#system_flag-2). Defaults to
+  whatever your system recommends, which is probably `65535`.
+
+
+### Internal options
+
+The following options are only used for testing nostrum itself.
+
+- `dev` - This is added to enable Nostrum to be run completely stand alone for
+  development purposes. `true` will cause Nostrum to spawn its own event
+  consumers. If you have the dev flag set to true while running Nostrum
+  alongside your application some of your events will be consumed. Defaults to
+  `false`.
 
 
 ## Logging

lib/nostrum/shard.ex

@@ -5,7 +5,7 @@ defmodule Nostrum.Shard do
 
   alias Nostrum.Shard.Session
 
-  def start_link([_, shard_num] = opts) do
+  def start_link([_, shard_num, _total] = opts) do
     Supervisor.start_link(__MODULE__, opts, name: :"Nostrum.Shard-#{shard_num}")
   end
 

lib/nostrum/shard/payload.ex

@@ -1,7 +1,7 @@
 defmodule Nostrum.Shard.Payload do
   @moduledoc false
 
-  alias Nostrum.{Constants, Shard.Intents, Util}
+  alias Nostrum.{Constants, Shard.Intents}
 
   @large_threshold 250
 
@@ -24,7 +24,7 @@ defmodule Nostrum.Shard.Payload do
       },
       "compress" => false,
       "large_threshold" => @large_threshold,
-      "shard" => [state.shard_num, Util.num_shards()],
+      "shard" => [state.shard_num, state.total_shards],
       "intents" => Intents.get_enabled_intents()
     }
     |> build_payload("IDENTIFY")

lib/nostrum/shard/session.ex

@@ -48,15 +48,17 @@ defmodule Nostrum.Shard.Session do
     GenServer.call(pid, :get_ws_state)
   end
 
-  def start_link([gateway, shard_num]) do
-    GenServer.start_link(__MODULE__, [gateway, shard_num], spawn_opt: [Util.fullsweep_after()])
+  def start_link([gateway, shard_num, total]) do
+    GenServer.start_link(__MODULE__, [gateway, shard_num, total],
+      spawn_opt: [Util.fullsweep_after()]
+    )
   end
 
-  def init([_gateway, _shard_num] = args) do
+  def init([_gateway, _shard_num, _total] = args) do
     {:ok, nil, {:continue, args}}
   end
 
-  def handle_continue([gateway, shard_num], nil) do
+  def handle_continue([gateway, shard_num, total_shards], nil) do
     Connector.block_until_connect()
     Logger.metadata(shard: shard_num)
 
@@ -73,6 +75,7 @@ defmodule Nostrum.Shard.Session do
       conn_pid: self(),
       conn: worker,
       shard_num: shard_num,
+      total_shards: total_shards,
       stream: stream,
       gateway: gateway <> @gateway_qs,
       last_heartbeat_ack: DateTime.utc_now(),

lib/nostrum/shard/supervisor.ex

@@ -44,33 +44,34 @@ defmodule Nostrum.Shard.Supervisor do
 
   require Logger
 
-  def start_link(_args) do
-    {url, gateway_shard_count} = Util.gateway()
-
-    num_shards =
-      case Application.get_env(:nostrum, :num_shards, :auto) do
-        :auto ->
-          gateway_shard_count
+  defp cast_shard_range(gateway_shards, :auto), do: {1, gateway_shards, gateway_shards}
+  defp cast_shard_range(gateway_shards, gateway_shards), do: {1, gateway_shards, gateway_shards}
+
+  defp cast_shard_range(gateway_shards, count) when is_integer(count) and count > 0 do
+    Logger.warn(
+      "Configured shard count (#{count}) " <>
+        "differs from Discord Gateway's recommended shard count (#{gateway_shards}). " <>
+        "Consider using `num_shards: :auto` option in your Nostrum config."
+    )
 
-        ^gateway_shard_count ->
-          gateway_shard_count
+    {1, count, count}
+  end
 
-        shard_count when is_integer(shard_count) and shard_count > 0 ->
-          Logger.warn(
-            "Configured shard count (#{shard_count}) " <>
-              "differs from Discord Gateway's recommended shard count (#{gateway_shard_count}). " <>
-              "Consider using `num_shards: :auto` option in your Nostrum config."
-          )
+  defp cast_shard_range(_gateway_shards, {lowest, highest, total} = range)
+       when is_integer(lowest) and is_integer(highest) and is_integer(total) and
+              lowest <= highest and highest <= total do
+    range
+  end
 
-          shard_count
+  def start_link(_args) do
+    {url, gateway_shard_count} = Util.gateway()
 
-        value ->
-          raise ~s("#{value}" is not a valid shard count)
-      end
+    value = Application.get_env(:nostrum, :num_shards, :auto)
+    shard_range = cast_shard_range(gateway_shard_count, value)
 
     Supervisor.start_link(
       __MODULE__,
-      [url, num_shards],
+      [url, shard_range],
       name: __MODULE__
     )
   end
@@ -101,16 +102,17 @@ defmodule Nostrum.Shard.Supervisor do
   end
 
   @doc false
-  def init([url, num_shards]) do
-    children = for i <- 0..(num_shards - 1), do: create_worker(url, i)
+  def init([url, {lowest, highest, total}]) do
+    shard_range = lowest..highest
+    children = for num <- shard_range, do: create_worker(url, num - 1, total)
 
     Supervisor.init(children, strategy: :one_for_one, max_restarts: 3, max_seconds: 60)
   end
 
   @doc false
-  def create_worker(gateway, shard_num) do
+  def create_worker(gateway, shard_num, total) do
     Supervisor.child_spec(
-      {Shard, [gateway, shard_num]},
+      {Shard, [gateway, shard_num, total]},
       id: shard_num
     )
   end

lib/nostrum/struct/ws_state.ex

@@ -5,6 +5,7 @@ defmodule Nostrum.Struct.WSState do
 
   defstruct [
     :shard_num,
+    :total_shards,
     :seq,
     :session,
     :shard_pid,
@@ -23,6 +24,15 @@ defmodule Nostrum.Struct.WSState do
   @typedoc "The shard number"
   @type shard_num :: non_neg_integer
 
+  @typedoc """
+  The highest shard number for this bot.
+
+  This may not be started locally, it is just used by nostrum to inform the
+  gateway which events we are interested in.
+  """
+  @typedoc since: "0.8.0"
+  @type total_shards :: non_neg_integer()
+
   @typedoc "The sequence number of the last event"
   @type seq :: integer | nil
 
@@ -71,6 +81,7 @@ defmodule Nostrum.Struct.WSState do
 
   @type t :: %__MODULE__{
           shard_num: shard_num,
+          total_shards: total_shards,
           seq: seq,
           session: session,
           shard_pid: shard_pid,

lib/nostrum/util.ex

@@ -53,19 +53,38 @@ defmodule Nostrum.Util do
   end
 
   @doc """
-  Returns the number of shards.
+  Returns the total amount of shards as per the configuration.
 
-  This is not the number of currently active shards, but the number of shards specified
-  in your config.
+  ## Return value
+
+  - If you specified your shards as `:auto`, the return value will be the
+  recommended number of shards as given by the gateway.
+
+  - If you explicitly specified your shard numbers as an integer, it will be
+  the given number.
+
+  - If you specified your shards in the form `{lowest, highest, total}` to
+  start a specific range of the total shards you want to start, this will be
+  the `total` value.
+
+  Should Discord not supply us with any shard information, this will return
+  `1`.
+
+  Note that this is not the number of currently active shards, but the number
+  of shards specified in your config.
   """
-  @spec num_shards() :: integer
+  @spec num_shards() :: pos_integer()
   def num_shards do
     num =
       with :auto <- Application.get_env(:nostrum, :num_shards, :auto),
-           {_url, shards} <- gateway(),
-           do: shards
+           {_url, shards} <- gateway() do
+        shards
+      end
 
-    if num == nil, do: 1, else: num
+    case num do
+      {_lowest, _highest, total} -> total
+      nil -> 1
+    end
   end
 
   @doc false