Memoize or precompute subgraphs that depend only on input shapes

[robertknight]

Many models have subgraphs that depend only on the shape of inputs, and thus don't change when the model is called repeatedly with inputs of the same shape. These subgraphs are usually cheap since the tensors flowing through them are small, but there is nevertheless overhead for each operation that is run. These subgraphs could be memoized to avoid re-running them unnecessarily.

As a starting point, it would be useful to do some experiments to see how many operations can be saved on some popular models, especially decoder models which are run repeatedly.