Use a single point to process non-differentiable functions in the reverse mode #1161
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Right now, we have a bunch of reasons to consider a function non-differentiable and not produce a pullback. In all cases, we clone the call and return early in different ways, almost all of which have flaws. For example:
Return on line
1488
: it's not necessary to pass 0 as the adjoint of non-differentiable entities in the reverse mode. Also, cloning arguments is dangerous for multiple reasons (e.g. sometimes we introduce operators, brackets, etc. which are not generated when cloning) and doesn't account for their possible differentiable side effects.Return on line
1520
: Again, cloning is dangerous.Return on line
1642
: Cloning again. Also, providing the cloned call as the adjoint doesn't make much sense.Return on line
1664
:dfdx()
of the call shouldn't be passed to its arguments.This PR replaces all of these cases to the same place and fixes the mentioned issues.