Move invalidate() to diff lib since it only deals with diffs

Move invalidate() to diff lib since it only deals with diffs #1382