activation intrinsics for neural networks

[jalvesz]

Add activation functions for neural networks. Relates to #858

Todo:

• specs & autodoc
• tests
• examples

Previous Art
SciPy Special functions collection: https://docs.scipy.org/doc/scipy/reference/special.html
torch nn: https://pytorch.org/docs/stable/nn.html#non-linear-activations-other
neural-fortran : https://github.com/modern-fortran/neural-fortran/blob/main/src/nf/nf_activation.f90

cc: @Beliavsky @milancurcic @epagone @jvdp1 @perazz

[jalvesz]

Before moving forward, any opinions on

• putting these functions within the "specialfunctions" category?
• naming of the derivative/gradient version with <name>_grad?
• any other remark?

[perazz]

Thank you for this implementation @jalvesz, I have left my comments. I am no machine learning expert, so I focused on the code rather than on the actual math of each function.

[perazz]

Please add a specs description of this function.

[perazz]

Please add a specs description for ftanh and ferf in the docs.

[perazz]

Also: I believe a discussion would be needed for fast, approximate versions of intrinsic functions. Different from gamma (that returns similar accuracy as the intrinsics), these are fast approximations. I'm fine with the f prefix (or fast_ would also be OK). I believe that should be standardized somewhere, for example in the library style guide?

[perazz]

Should the tests be run for all accuracies?

[jalvesz]

Thanks @perazz for your review! regarding tanh and erf, I wonder if I should actually remove reference to the intrinsic names and simply leave a reference to the fast approximation as that is what makes sense for NNs. Also, should these functions stay here or moved to the intrinsics module?

Regarding

I'm fine with the f prefix (or fast_ would also be OK). I believe that should be standardized somewhere, for example in the library style guide?

I agree with you and also wonder which one would be preferable, I don't have strong oppinions on that.