-
Notifications
You must be signed in to change notification settings - Fork 122
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Compute and process Differentiation Request graph
Plan for dynamic graph - The relations between different differentiation requests can be modelled as a graph. For example, if `f_a` calls `f_b`, there will be two differentiation requests `df_a` and `df_b`, the edge between them can be understood as `created_because_of`. This also means that the functions called by the users to be explicitly differentiated (or `DiffRequests` created because of these) are the source nodes, i.e. no incoming edges. In most cases, this graph aligns with the call graph, but in some cases, the graph depends on the internal implementation, like the Hessian computation, which requires creating multiple `fwd_mode` requests followed by a `rev_mode` request. - We can use this graph to order the computation of differentiation requests. This was already being done implicitly in the initial recursive implementation. Whenever we encountered a call expression, we started differentiation of the called function; this was sort of like a depth-first search strategy. - This had problems, as `Clang` reported errors when it encountered a new function scope (of the derivative of the called function) in the middle of the old function scope (of the derivative of the callee function). It treated the nested one like a lambda expression. The issue regarding this: #745. - To fix this, an initial strategy was to eliminate the recursive approach. Hence, a queue-based breadth-first approach was implemented in this PR: #848. - Although it fixed the problem, the graph traversal was still implicit. We needed some way to compute/store the complete graph and possibly optimize it, such as converting edges to model the `requires_derivative_of` relation. Using this, we could proceed with differentiation in a topologically sorted ordering. - It also required one caveat: although we don't differentiate the called function completely in a recursive way, we still need to declare it so that we can have the call expression completed (i.e. `auto t0 = f_pushforward(...)`). - To move towards the final stage of having the complete graph computed before starting the differentiation, we need the complete information on how the `DiffRequest` will be formed inside the visitors (including arguments or `DVI` info). This whole approach will require activity analysis in the first pass. - As an incremental improvement, the first requirement was to implement infrastructure to support explicit modelling of the graph and use that to have a breadth-first traversal (and eventually topological ordering). This is the initial PR for capturing the differentiation plan in a graphical format. However, the traversal order is still breadth-first, as we don't have the complete graph in the first pass - mainly because of a lack of information about the args required for `pushforward` and `pullbacks`. This can be improved with the help of activity analysis to capture the complete graph in the first pass, processing the plan in a topologically sorted manner and pruning the graph for user-defined functions. I started this with this approach, and the initial experimental commit is available here for future reference: vaithak@82c0b42.
- Loading branch information
1 parent
922bd1c
commit b59c9aa
Showing
16 changed files
with
398 additions
and
67 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| #ifndef CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H | ||
| #define CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H | ||
|
|
||
| #include <algorithm> | ||
| #include <functional> | ||
| #include <iostream> | ||
| #include <queue> | ||
| #include <set> | ||
| #include <unordered_map> | ||
| #include <unordered_set> | ||
| #include <vector> | ||
|
|
||
| namespace clad { | ||
| template <typename T> class DynamicGraph { | ||
| private: | ||
| /// Storing nodes in the graph. The index of the node in the vector is used as | ||
| /// a unique identifier for the node in the adjacency list. | ||
| std::vector<T> m_nodes; | ||
|
|
||
| /// Store the nodes in the graph as an unordered map from the node to a | ||
| /// boolean indicating whether the node is processed or not. The second | ||
| /// element in the pair is the id of the node in the nodes vector. | ||
| std::unordered_map<T, std::pair<bool, size_t>> m_nodeMap; | ||
|
|
||
| /// Store the adjacency list for the graph. The adjacency list is a map from | ||
| /// a node to the set of nodes that it has an edge to. We use integers inside | ||
| /// the set to avoid copying the nodes. | ||
| std::unordered_map<size_t, std::set<size_t>> m_adjList; | ||
|
|
||
| /// Set of source nodes in the graph. | ||
| std::set<size_t> m_sources; | ||
|
|
||
| /// Store the id of the node being processed right now. | ||
| int m_currentId = -1; // -1 means no node is being processed. | ||
|
|
||
| /// Maintain a queue of nodes to be processed next. | ||
| std::queue<size_t> m_toProcessQueue; | ||
|
|
||
| public: | ||
| DynamicGraph() = default; | ||
|
|
||
| /// Add an edge from the source node to the destination node. | ||
| /// \param src | ||
| /// \param dest | ||
| void addEdge(const T& src, const T& dest) { | ||
| std::pair<bool, size_t> srcInfo = addNode(src); | ||
| std::pair<bool, size_t> destInfo = addNode(dest); | ||
| size_t srcId = srcInfo.second; | ||
| size_t destId = destInfo.second; | ||
| m_adjList[srcId].insert(destId); | ||
| } | ||
|
|
||
| /// Add a node to the graph. If the node is already present, return the | ||
| /// id of the node in the graph. If the node is a source node, add it to the | ||
| /// queue of nodes to be processed. | ||
| /// \param node | ||
| /// \param isSource | ||
| /// \returns A pair of a boolean indicating whether the node is already | ||
| /// processed and the id of the node in the graph. | ||
| std::pair<bool, size_t> addNode(const T& node, bool isSource = false) { | ||
| if (m_nodeMap.find(node) == m_nodeMap.end()) { | ||
| size_t id = m_nodes.size(); | ||
| m_nodes.push_back(node); | ||
| m_nodeMap[node] = {false, id}; // node is not processed yet. | ||
| m_adjList[id] = {}; | ||
| if (isSource) { | ||
| m_sources.insert(id); | ||
| m_toProcessQueue.push(id); | ||
| } | ||
| } | ||
| return m_nodeMap[node]; | ||
| } | ||
|
|
||
| /// Add an edge from the current node being processed to the | ||
| /// destination node. | ||
| /// \param dest | ||
| void addEdgeToCurrentNode(const T& dest) { | ||
| if (m_currentId == -1) | ||
| return; | ||
| addEdge(m_nodes[m_currentId], dest); | ||
| } | ||
|
|
||
| /// Set the current node being processed. | ||
| /// \param node | ||
| void setCurrentProcessingNode(const T& node) { | ||
| if (m_nodeMap.find(node) != m_nodeMap.end()) | ||
| m_currentId = m_nodeMap[node].second; | ||
| } | ||
|
|
||
| /// Mark the current node being processed as processed and add the | ||
| /// destination nodes to the queue of nodes to be processed. | ||
| void markCurrentNodeProcessed() { | ||
| if (m_currentId != -1) { | ||
| m_nodeMap[m_nodes[m_currentId]].first = true; | ||
| for (size_t destId : m_adjList[m_currentId]) | ||
| if (!m_nodeMap[m_nodes[destId]].first) | ||
| m_toProcessQueue.push(destId); | ||
| } | ||
| m_currentId = -1; | ||
| } | ||
|
|
||
| /// Get the nodes in the graph. | ||
| std::vector<T> getNodes() { return m_nodes; } | ||
|
|
||
| /// Print the nodes and edges in the graph. | ||
| void print() { | ||
| // First print the nodes with their insertion order. | ||
| for (const T& node : m_nodes) { | ||
| std::pair<bool, int> nodeInfo = m_nodeMap[node]; | ||
| std::cout << (std::string)node << ": #" << nodeInfo.second; | ||
| if (m_sources.find(nodeInfo.second) != m_sources.end()) | ||
| std::cout << " (source)"; | ||
| if (nodeInfo.first) | ||
| std::cout << ", (done)\n"; | ||
| else | ||
| std::cout << ", (unprocessed)\n"; | ||
| } | ||
| // Then print the edges. | ||
| for (int i = 0; i < m_nodes.size(); i++) | ||
| for (size_t dest : m_adjList[i]) | ||
| std::cout << i << " -> " << dest << "\n"; | ||
| } | ||
|
|
||
| /// Get the next node to be processed from the queue of nodes to be | ||
| /// processed. | ||
| /// \returns The next node to be processed. | ||
| T getNextToProcessNode() { | ||
| if (m_toProcessQueue.empty()) | ||
| return T(); | ||
| size_t nextId = m_toProcessQueue.front(); | ||
| m_toProcessQueue.pop(); | ||
| return m_nodes[nextId]; | ||
| } | ||
| }; | ||
| } // end namespace clad | ||
|
|
||
| #endif // CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.