-
Notifications
You must be signed in to change notification settings - Fork 123
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Compute and process Differentiation Request graph
Plan for dynamic graph - The relations between different differentiation requests can be modelled as a graph. For example, if `f_a` calls `f_b`, there will be two differentiation requests `df_a` and `df_b`, the edge between them can be understood as `created_because_of`. This also means that the functions called by the users to be explicitly differentiated (or `DiffRequests` created because of these) are the source nodes, i.e. no incoming edges. In most cases, this graph aligns with the call graph, but in some cases, the graph depends on the internal implementation, like the Hessian computation, which requires creating multiple `fwd_mode` requests followed by a `rev_mode` request. - We can use this graph to order the computation of differentiation requests. This was already being done implicitly in the initial recursive implementation. Whenever we encountered a call expression, we started differentiation of the called function; this was sort of like a depth-first search strategy. - This had problems, as `Clang` reported errors when it encountered a new function scope (of the derivative of the called function) in the middle of the old function scope (of the derivative of the callee function). It treated the nested one like a lambda expression. The issue regarding this: #745. - To fix this, an initial strategy was to eliminate the recursive approach. Hence, a queue-based breadth-first approach was implemented in this PR: #848. - Although it fixed the problem, the graph traversal was still implicit. We needed some way to compute/store the complete graph and possibly optimize it, such as converting edges to model the `requires_derivative_of` relation. Using this, we could proceed with differentiation in a topologically sorted ordering. - It also required one caveat: although we don't differentiate the called function completely in a recursive way, we still need to declare it so that we can have the call expression completed (i.e. `auto t0 = f_pushforward(...)`). - To move towards the final stage of having the complete graph computed before starting the differentiation, we need the complete information on how the `DiffRequest` will be formed inside the visitors (including arguments or `DVI` info). This whole approach will require activity analysis in the first pass. - As an incremental improvement, the first requirement was to implement infrastructure to support explicit modelling of the graph and use that to have a breadth-first traversal (and eventually topological ordering). This is the initial PR for capturing the differentiation plan in a graphical format. However, the traversal order is still breadth-first, as we don't have the complete graph in the first pass - mainly because of a lack of information about the args required for `pushforward` and `pullbacks`. This can be improved with the help of activity analysis to capture the complete graph in the first pass, processing the plan in a topologically sorted manner and pruning the graph for user-defined functions. I started this with this approach, and the initial experimental commit is available here for future reference: vaithak@82c0b42.
- Loading branch information
Showing
12 changed files
with
410 additions
and
67 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| #ifndef CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H | ||
| #define CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H | ||
|
|
||
| #include <algorithm> | ||
| #include <functional> | ||
| #include <iostream> | ||
| #include <queue> | ||
| #include <set> | ||
| #include <unordered_map> | ||
| #include <unordered_set> | ||
| #include <vector> | ||
|
|
||
| namespace clad { | ||
| template <typename T> class DynamicGraph { | ||
| private: | ||
| // Storing nodes in the graph. The index of the node in the vector is used as | ||
| // a unique identifier for the node in the adjacency list. | ||
| std::vector<T> m_nodes; | ||
|
|
||
| // Store the nodes in the graph as an unordered map from the node to a boolean | ||
| // indicating whether the node is processed or not. The second element in the | ||
| // pair is the id of the node in the nodes vector. | ||
| std::unordered_map<T, std::pair<bool, size_t>> m_nodeMap; | ||
|
|
||
| // Store the adjacency list for the graph. The adjacency list is a map from | ||
| // a node to the set of nodes that it has an edge to. We use integers inside | ||
| // the set to avoid copying the nodes. | ||
| std::unordered_map<size_t, std::set<size_t>> m_adjList; | ||
|
|
||
| // Set of source nodes in the graph. | ||
| std::set<size_t> m_sources; | ||
|
|
||
| // Store the id of the node being processed right now. | ||
| int m_currentId = -1; // -1 means no node is being processed. | ||
|
|
||
| // Maintain a queue of nodes to be processed next. | ||
| std::queue<size_t> m_toProcessQueue; | ||
|
|
||
| public: | ||
| DynamicGraph() = default; | ||
|
|
||
| // Add an edge from src to dest | ||
| void addEdge(const T& src, const T& dest) { | ||
| std::pair<bool, size_t> srcInfo = addNode(src); | ||
| std::pair<bool, size_t> destInfo = addNode(dest); | ||
| size_t srcId = srcInfo.second; | ||
| size_t destId = destInfo.second; | ||
| m_adjList[srcId].insert(destId); | ||
| } | ||
|
|
||
| // Add a node to the graph | ||
| std::pair<bool, size_t> addNode(const T& node, bool isSource = false) { | ||
| if (m_nodeMap.find(node) == m_nodeMap.end()) { | ||
| size_t id = m_nodes.size(); | ||
| m_nodes.push_back(node); | ||
| m_nodeMap[node] = {false, id}; // node is not processed yet. | ||
| m_adjList[id] = {}; | ||
| if (isSource) { | ||
| m_sources.insert(id); | ||
| m_toProcessQueue.push(id); | ||
| } | ||
| } | ||
| return m_nodeMap[node]; | ||
| } | ||
|
|
||
| // Adds the edge from the current node to the destination node. | ||
| void addEdgeToCurrentNode(const T& dest) { | ||
| if (m_currentId == -1) | ||
| return; | ||
| addEdge(m_nodes[m_currentId], dest); | ||
| } | ||
|
|
||
| // Set the current node to the node with the given id. | ||
| void setCurrentProcessingNode(const T& node) { | ||
| if (m_nodeMap.find(node) != m_nodeMap.end()) | ||
| m_currentId = m_nodeMap[node].second; | ||
| } | ||
|
|
||
| // Mark the current node as processed. | ||
| void markCurrentNodeProcessed() { | ||
| if (m_currentId != -1) { | ||
| m_nodeMap[m_nodes[m_currentId]].first = true; | ||
| for (size_t destId : m_adjList[m_currentId]) | ||
| if (!m_nodeMap[m_nodes[destId]].first) | ||
| m_toProcessQueue.push(destId); | ||
| } | ||
| m_currentId = -1; | ||
| } | ||
|
|
||
| // Get the nodes in the graph. | ||
| std::vector<T> getNodes() { return m_nodes; } | ||
|
|
||
| // Check if two nodes are connected in the graph. | ||
| bool isConnected(const T& src, const T& dest) { | ||
| if (m_nodeMap.find(src) == m_nodeMap.end() || | ||
| m_nodeMap.find(dest) == m_nodeMap.end()) | ||
| return false; | ||
| size_t srcId = m_nodeMap[src].second; | ||
| size_t destId = m_nodeMap[dest].second; | ||
| return m_adjList[srcId].find(destId) != m_adjList[srcId].end(); | ||
| } | ||
|
|
||
| // Print the graph in a human-readable format. | ||
| void print() { | ||
| // First print the nodes with their insertion order. | ||
| for (const T& node : m_nodes) { | ||
| std::pair<bool, int> nodeInfo = m_nodeMap[node]; | ||
| std::cout << (std::string)node << ": #" << nodeInfo.second; | ||
| if (m_sources.find(nodeInfo.second) != m_sources.end()) | ||
| std::cout << " (source)"; | ||
| if (nodeInfo.first) | ||
| std::cout << ", (done)\n"; | ||
| else | ||
| std::cout << ", (unprocessed)\n"; | ||
| } | ||
| // Then print the edges. | ||
| for (int i = 0; i < m_nodes.size(); i++) | ||
| for (size_t dest : m_adjList[i]) | ||
| std::cout << i << " -> " << dest << "\n"; | ||
| } | ||
|
|
||
| // Topological sort of the directed graph. If the graph is not a DAG, the | ||
| // result will be a partial order. Use a recursive dfs heler function to | ||
| // implement the topological sort. If a->b, then a will come before b in the | ||
| // topological sort. In reverseOrder mode, the result will be in reverse | ||
| // topological order, i.e a->b, then b will come before a in the result. | ||
| std::vector<T> topologicalSort(bool reverseOrder = false) { | ||
| std::vector<T> res; | ||
| std::unordered_set<size_t> visited; | ||
|
|
||
| std::function<void(size_t)> dfs = [&](size_t node) -> void { | ||
| visited.insert(node); | ||
| for (size_t dest : m_adjList[node]) | ||
| if (visited.find(dest) == visited.end()) | ||
| dfs(dest); | ||
| res.push_back(m_nodes[node]); | ||
| }; | ||
| for (size_t source : m_sources) | ||
| if (visited.find(source) == visited.end()) | ||
| dfs(source); | ||
|
|
||
| if (reverseOrder) | ||
| return res; | ||
| std::reverse(res.begin(), res.end()); | ||
| return res; | ||
| } | ||
|
|
||
| // Get the next to process node from the queue of nodes to be processed. | ||
| T getNextToProcessNode() { | ||
| if (m_toProcessQueue.empty()) | ||
| return T(); | ||
| size_t nextId = m_toProcessQueue.front(); | ||
| m_toProcessQueue.pop(); | ||
| return m_nodes[nextId]; | ||
| } | ||
| }; | ||
| } // end namespace clad | ||
|
|
||
| #endif // CLAD_DIFFERENTIATOR_DYNAMICGRAPH_H |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.