Use faster activation functions

[mcabbott]

This substitutes tanh_fast where possible, since tanh often dominates the forward pass of small networks. Builds on #1761, but in the months that sat waiting, I have mislaid the benchmarks I ran. Best case 2x better forwards, worst case no improvement, modulo noise.

Intersects with #1832, which would also do this to conv layers, but not to RNNs.

Closes #1272; this version is significantly faster, and this PR applies it to more cases.

[CarloLucibello]

It's a pity to pollute the code a bit but I guess the performance increase is worth it.

I wonder what could be a more aesthetically pleasing alternative. Pointing NNlib.sigmod to sigmoid_fast and defining a NNlib.tanh as tanh_fast?

Just for reference, the reason why this PR is switching activation in the forward instead of construction time is here

My thinking is that we may later decide that it's better to skip tanh_fast on the GPU. I can't measure a difference so who knows. To do that, we can add a method in NNlibCUDA like fast_act(::typeof(tanh), ::CuArray) = tanh. Provided you call this like fast = NNlib.fast_act(fun, x) with an example of the array you plan to broadcast it over.

[mcabbott]

I wouldn't claim this as an aesthetic improvement. But it is a performance one.

I'm against less explicit names like NNlib.tanh, I think it's confusing to read packages which don't work how they look like they work, because some Base name means something different. Flux.rand used to be some such trick, and we ripped it out, for explicit rand32.

[CarloLucibello]

Ok. Should we wait for a final tag of the v0.12 before merging (#1838)? I wouldn't say this PR is breaking though

[mcabbott]

Not so breaking as to demand a release, but if there's one nearby perhaps that's the safe time. So after last 0.12.x tag, before v0.13, as you say.

[ToucheSir]

Buildkite is failing with weird errors again 😬

[ToucheSir]

bors try

[bors]

try

Build succeeded:

• buildkite/flux-dot-jl

[codecov-commenter]

Codecov Report

Merging #1837 (0a8fade) into master (8d3b8d3) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1837      +/-   ##
==========================================
+ Coverage   73.85%   73.94%   +0.09%     
==========================================
  Files          28       28              
  Lines        1683     1689       +6     
==========================================
+ Hits         1243     1249       +6     
  Misses        440      440              

Impacted Files	Coverage Δ
src/layers/basic.jl	75.00% <100.00%> (+0.19%)	⬆️
src/layers/conv.jl	80.11% <100.00%> (+0.44%)	⬆️
src/layers/recurrent.jl	75.67% <100.00%> (+0.22%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d3b8d3...0a8fade. Read the comment docs.

[ToucheSir]

Ought these to get the fast_act treatment too? I'm ok with revisiting too, RNN cell inflexibility is a bit of a long standing issue.

[mcabbott]

Oh right. If we decide to disable fast_tanh on CuArrays, that will be ignored here. But perhaps revisit if & when... it's a bit more clutter to squeeze that in.

[ToucheSir]

It may be a blessing in disguise, as currently plain tanh.(...) will hit the likely obsolete https://github.com/FluxML/NNlibCUDA.jl/blob/master/src/cudnn/activations.jl.