Unclear wording in "Composing Optimizers" section of docs

[StevenWhitaker]

In the Composing Optimizers section of the docs it states:

opt = Optimiser(ExpDecay(0.001, 0.1, 1000, 1e-4), Descent())

Here we apply exponential decay to the Descent optimiser.

I think that last sentence is a bit misleading to someone who is not very familiar with Flux. When I read "apply exponential decay to the Descent optimiser", I take that to mean that we use the learning rate defined by Descent, but update it according to some schedule defined by ExpDecay. But in reality, if I understand correctly, the above example from the docs uses an initial learning rate that is actually 0.0001 (because the default learning rate of Descent is 0.1), not 0.1 like I would expect.

(Somewhat tangential note: One possible reason for the confusion might be the fact that ExpDecay by itself can be used to do gradient descent, but that isn't conveyed by the name ExpDecay. I would think that something called ExpDecay would have to be paired with an optimizer, and would not have its own learning rate.)

Two possible ways to improve the clarity of the docs:

1. Update the example with

   opt = Optimiser(ExpDecay(1, 0.1, 1000, 1e-4), Descent())

   and leave the rest of the wording as is.

2. Use a different example, especially since

   opt = Optimiser(ExpDecay(0.001, 0.1, 1000, 1e-4), Descent())

   is equivalent to

   opt = ExpDecay(0.0001, 0.1, 1000, 1e-4)

   (if I understand correctly), so the current example doesn't seem as useful as another might be.

I am happy to open a PR to make the simple change suggested in 1., but I don't know Flux well enough to do 2.

[darsnack]

Short term fix: perhaps the default value for the LR in ExpDecay should be 1 so it composes sensibly with other optimizers.

Long term rant:

One possible reason for the confusion might be the fact that ExpDecay by itself can be used to do gradient descent, but that isn't conveyed by the name ExpDecay. I would think that something called ExpDecay would have to be paired with an optimizer, and would not have its own learning rate.

Thank you for taking the effort to type this up, because what you are highlighting here is something that I've brought up before. Schedules and optimizers are not the same thing. It's a cute trick that certain schedules can be "composed" with our optimizers, but I really strongly feel we should fix this by making schedules their own distinct thing.

Schedulers (in general) wrap an optimizer, they do not compose with them. The fact that ExpDecay can currently be used like an optimizer when it isn't one just underscores this point so well.

[DhairyaLGandhi]

1 should be fine for now