Emily Boudreaux
6b57738330
Note for now much of the docs is LLM written. This was done to save time and focus on code development. The docs have been LOOSLEY reviewed. However, the code itself has been either fully written by hand or fully reviewed. Therefore, the docs should at the moment be seen as a loose resource rather than authoratiative (and this is on top of the general ability for docs to drift)
174 lines
15 KiB
TeX
174 lines
15 KiB
TeX
\doxysection{src/include/serif/utils/dependencies/dependencies.hpp File Reference}
|
|
\hypertarget{utils_2dependencies_2dependencies_8hpp}{}\label{utils_2dependencies_2dependencies_8hpp}\index{src/include/serif/utils/dependencies/dependencies.hpp@{src/include/serif/utils/dependencies/dependencies.hpp}}
|
|
|
|
|
|
This file is a convenience header which includes all of the headers in the dependencies module. Further, we include the detailed documentation for the dependencies module here.
|
|
|
|
|
|
{\ttfamily \#include "{}serif/utils/dependencies/keys.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/binding.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/key\+\_\+list.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/key\+\_\+list\+\_\+operations.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/dependency\+\_\+context.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/computation.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/incremental\+\_\+computation.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/computation\+\_\+graph.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/computation\+\_\+cache.\+hpp"{}}\newline
|
|
{\ttfamily \#include "{}serif/utils/dependencies/snapshot.\+hpp"{}}\newline
|
|
Include dependency graph for dependencies.\+hpp\+:
|
|
\nopagebreak
|
|
\begin{figure}[H]
|
|
\begin{center}
|
|
\leavevmode
|
|
\includegraphics[width=350pt]{utils_2dependencies_2dependencies_8hpp__incl}
|
|
\end{center}
|
|
\end{figure}
|
|
This graph shows which files directly or indirectly include this file\+:
|
|
\nopagebreak
|
|
\begin{figure}[H]
|
|
\begin{center}
|
|
\leavevmode
|
|
\includegraphics[width=350pt]{utils_2dependencies_2dependencies_8hpp__dep__incl}
|
|
\end{center}
|
|
\end{figure}
|
|
|
|
|
|
\doxysubsection{Detailed Description}
|
|
This file is a convenience header which includes all of the headers in the dependencies module. Further, we include the detailed documentation for the dependencies module here.
|
|
|
|
\begin{DoxyAuthor}{Author}
|
|
Emily M. Boudreaux
|
|
\end{DoxyAuthor}
|
|
{\itshape Dependencies Module}
|
|
|
|
Demonstration code for how to use the SERiF computational graph system The broad idea here is that there are many calculations which depend on each other but which may not necessarily always need to be recomputed just because another part of the model has changed. We model this as a combination of a acyclic computational graph, defined at compile time, and a runtime cache of values. The general workflow is that a developer defines the "{}context"{} of the computational graph, think of this as all of the inputs to the graph, or maybe more specifically, anything that could change due to some external event (e.\+g. a time step, a newton step, etc...) This context is probably the most CS intensive part of this code as the developer needs to think about lifetimes and ownership.
|
|
|
|
{\bfseries{Context}} Contexts are made with the make\+\_\+dependency\+\_\+context function, this accepts a variadic pack of bindings between a type and a runtime value. You can generate these yourself using the binding functions and types; however, we provide two convince wrappers which generally developers should consider using.
|
|
\begin{DoxyItemize}
|
|
\item borrow\+: Used to bind a type to a reference to a value. This is useful for values which are owned by the caller and which the caller will manage the lifetime of (takes an lvalue).
|
|
\item own\+: Used to bind a type to a value which is owned by the context (takes an rvalue) Each of these functions accepts a type as a template parameter and a runtime value as an argument. The type is used as an ID, this is key to understand, if we write
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{\textcolor{keyword}{using\ namespace\ }\mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies}{serif::utils::dependencies}};}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ x\{1.0\};}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ context\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab0cfd959e8c00f5e1641f26798333972}{make\_dependency\_context}}(\mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_abdacec2dea2bb64ecda1946529961565}{borrow<keys::Input>}}(x));}
|
|
|
|
\end{DoxyCode}
|
|
|
|
\end{DoxyItemize}
|
|
|
|
we are not saying that x is of type keys\+::\+Input, this is clearly not the case. Rather, we are registering in the computational graph a binding between the type keys\+::\+Input and the runtime value x. Think of this as naming the various inputs to the graph.
|
|
|
|
{\bfseries{Computations}} After defining a context we can define a computation, computations define both nodes and edges in the computational graph (nodes by the body, edges by the template parameters). Consider the following example (assuming the context from above)\+:
|
|
|
|
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_a70e11340fa044443fefa79ba7e0c11d5}{make\_computation<keys::Output,\ InputList<keys::Input>}}>(}
|
|
\DoxyCodeLine{\ [](\textcolor{keyword}{const}\ \textcolor{keywordtype}{double}\ \_x)\ -\/>\ \textcolor{keywordtype}{double}\ \{}
|
|
\DoxyCodeLine{\ \ \ \ \ \textcolor{keywordflow}{return}\ \_x\ *\ \_x;}
|
|
\DoxyCodeLine{\ \}}
|
|
\DoxyCodeLine{)}
|
|
|
|
\end{DoxyCode}
|
|
|
|
|
|
what we have defined here is a node which takes a single input tagged keys\+::\+Input and produces an output tagged as keys\+::\+Output. Note how we have implicitly registerd a new edge on the graph (even though its not in the context). This is an internal edge which cannot be accessed by the user. Rather, these internal edges allow us to connect computations together to form a larger graph. Consider now the following code
|
|
|
|
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{\textcolor{keyword}{using\ namespace\ }\mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies}{serif::utils::dependencies}};}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keyword}{struct\ }tags\ \{}
|
|
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{struct\ }X\{\};}
|
|
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{struct\ }Y\{\};}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{struct\ }Output1\{\};}
|
|
\DoxyCodeLine{\ \ \ \ \textcolor{keyword}{struct\ }Output2\{\};}
|
|
\DoxyCodeLine{\};}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ x\{1.0\};}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ y\{2.0\};}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ context\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab0cfd959e8c00f5e1641f26798333972}{make\_dependency\_context}}(\mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_abdacec2dea2bb64ecda1946529961565}{borrow<tags::X>}}(x),\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_abdacec2dea2bb64ecda1946529961565}{borrow<tags::Y>}}(y));}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func1\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_a70e11340fa044443fefa79ba7e0c11d5}{make\_computation<tags::Output1,\ InputList<tags::X>}}>(}
|
|
\DoxyCodeLine{\ \ \ \ [](\textcolor{keyword}{const}\ \textcolor{keywordtype}{double}\ \_x)\ -\/>\ \textcolor{keywordtype}{double}\ \{}
|
|
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \_x\ *\ \_x;}
|
|
\DoxyCodeLine{\ \ \ \ \}}
|
|
\DoxyCodeLine{)}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func2\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_a70e11340fa044443fefa79ba7e0c11d5}{make\_computation}}<tags::Output2,\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_a22eef682b02dad9de4774e4b68b003ba}{InputList<tags::Output1,\ tags::Y>}}>(}
|
|
\DoxyCodeLine{\ \ \ [](\textcolor{keyword}{const}\ \textcolor{keywordtype}{double}\ \_x,\ \textcolor{keyword}{const}\ \textcolor{keywordtype}{double}\ \_y)\ -\/>\ \textcolor{keywordtype}{double}\ \{}
|
|
\DoxyCodeLine{\ \ \ \ \ \ \ \ \textcolor{keywordflow}{return}\ \_x\ +\ \_y;}
|
|
\DoxyCodeLine{\ \ \ \ \}}
|
|
\DoxyCodeLine{);}
|
|
|
|
\end{DoxyCode}
|
|
|
|
|
|
we have now defined two computations, the first takes a single input, whichever input is tagged as tags\+::X and produces an output tagged as tags\+::\+Output1. The second computation takes two inputs, the first is the output of the first computation (lookup by the tag) and the second is whichever input is tagged as tags\+::Y, and produces an output tagged as tags\+::\+Output2. Note that the order of arguments to the lambda must be the same as the order of the tags in the Input\+List, the dependencies module will automatically order arguments based on the Input\+List. There is type checking to ensure that the types match, but if you have two inputs of the same type that are flipped there is no way for the system to inspect that and warn you. Further, depending on your compiler and flags there may be implicit casting between convertable types (e.\+g. int \texorpdfstring{$<$}{<}-\/\texorpdfstring{$>$}{>} double).
|
|
|
|
\begin{DoxyRefDesc}{Todo}
|
|
\item[\mbox{\hyperlink{todo__todo000006}{Todo}}]We may want to remove the ability to have implicit casting between types, this is a bit of a footgun and may lead to subtle bugs.\end{DoxyRefDesc}
|
|
|
|
|
|
\begin{DoxyNote}{Note}
|
|
This is primary goal of this system, split computations into small pieces with the ability to cache intermediate results and reuse as many of those intermediate results as we can can prove is safe to do. The whole reason this system is so complex is that very purpose. There is a fundamental problem with high dimensional stellar models, they are always going to be slow, therefore we need to accept some degree of complexity in order to make them tractable.
|
|
\end{DoxyNote}
|
|
{\bfseries{Computation Cache}} Once we have the computations we need to connect them together, generally use the helper function make\+\_\+computational\+\_\+cache which takes a context and a variadic pack of computations. There are {\itshape a lot} of compile time checks which run to ensure that the graph is valid, the context describes what is needed to compute the graph, and that each computation is valid. Generally if a static analysis error or compile error pops up here it probably means there is a mistake in the definition of the context or the computations. I\textquotesingle{}ve tried to make these errors as clear as possible; however, there is only so much that you can do with the C++ compiler. I recommend first checking, probably with a drawn-\/out diagram, that all the tags are correct and what you expect.
|
|
|
|
A developer can also choose to use the make\+\_\+computational\+\_\+graph function which provides a bit more explicit control; however, there is generally very little reason to do this and that function is primarily intended for internal use (for those interested make\+\_\+computational\+\_\+cache just calls make\+\_\+computational\+\_\+graph and then binds the particular context to the graph). Now consider the following code assuming the context and computations from above\+:
|
|
|
|
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ graph\_cache\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_a1aeb221a544c343e6f98e7889a6ee305}{make\_computation\_cache}}(graph\_context,\ quadratic\_function,\ linear\_function);}
|
|
|
|
\end{DoxyCode}
|
|
|
|
|
|
At this point we have a cache which we can access various parts of; however, no computations have been run yet. To run a computation we must call try\+\_\+prepare.
|
|
|
|
\begin{DoxyRefDesc}{Todo}
|
|
\item[\mbox{\hyperlink{todo__todo000007}{Todo}}]We really should remove the try\+\_\+prepare\textquotesingle{}dness of it all, that function uses exceptions for runtime logic which would make Herb Sutter very mad (and on a more serious note can lead to significant performance issues as the stack is unwound). This is a larger refactor as we need to propegate value based error handeling all through the code but should be done at some point.\end{DoxyRefDesc}
|
|
|
|
|
|
try\+\_\+prepare takes the context and attempts to evaluate the graph using the minimum number of nodal evaluations necessary. This means the first time, since nothing has been cached yet, the entire graph will be evaluated.
|
|
|
|
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ report\ =\ graph\_cache.try\_prepare(graph\_context);}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ func1\_result\ =\ graph\_cache.template\ get<tags::Output1>();}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ func2\_result\ =\ graph\_cache.template\ get<tags::Output2>();}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func1\_status\ =\ report-\/>template\ status<tags::Output1>();}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func2\_status\ =\ report-\/>template\ status<tags::Output2>();}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{std::string\_view\ func1\_status\_string\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab3b86a8b49e4a73b04c3fa547b45f8b2}{preparation\_status\_name\_map}}[func1\_status];\ \textcolor{comment}{//\ note\ this\ is\ an\ inline\ static\ map\ defined\ in\ serif::utils::dependencies}}
|
|
\DoxyCodeLine{std::string\_view\ func2\_status\_string\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab3b86a8b49e4a73b04c3fa547b45f8b2}{preparation\_status\_name\_map}}[func2\_status];}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{std::println(\textcolor{stringliteral}{"{}Func1\ Result:\ \{\},\ Func2\ Result:\ \{\},\ Func1\ Status:\ \{\},\ Func2\ Status:\ \{\}"{}},\ func1\_result,\ func2\_result,\ func1\_status\_string,\ func2\_status\_string);}
|
|
|
|
\end{DoxyCode}
|
|
|
|
|
|
Now what we could do is change only one part of the input context and call try\+\_\+prepare again. Lets say we change only y, such that only func2 is stale.
|
|
|
|
|
|
\begin{DoxyCode}{0}
|
|
\DoxyCodeLine{y\ =\ 3.0;}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ report\ =\ graph\_cache.try\_prepare(graph\_context);}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ func1\_result\ =\ graph\_cache.template\ get<tags::Output1>();}
|
|
\DoxyCodeLine{\textcolor{keywordtype}{double}\ func2\_result\ =\ graph\_cache.template\ get<tags::Output2>();}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func1\_status\ =\ report-\/>template\ status<tags::Output1>();}
|
|
\DoxyCodeLine{\textcolor{keyword}{auto}\ func2\_status\ =\ report-\/>template\ status<tags::Output2>();}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{std::string\_view\ func1\_status\_string\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab3b86a8b49e4a73b04c3fa547b45f8b2}{preparation\_status\_name\_map}}[func1\_status];\ \textcolor{comment}{//\ note\ this\ is\ an\ inline\ static\ map\ defined\ in\ serif::utils::dependencies}}
|
|
\DoxyCodeLine{std::string\_view\ func2\_status\_string\ =\ \mbox{\hyperlink{namespaceserif_1_1utils_1_1dependencies_ab3b86a8b49e4a73b04c3fa547b45f8b2}{preparation\_status\_name\_map}}[func2\_status];}
|
|
\DoxyCodeLine{}
|
|
\DoxyCodeLine{std::println(\textcolor{stringliteral}{"{}Func1\ Result:\ \{\},\ Func2\ Result:\ \{\},\ Func1\ Status:\ \{\},\ Func2\ Status:\ \{\}"{}},\ func1\_result,\ func2\_result,\ func1\_status\_string,\ func2\_status\_string);}
|
|
|
|
\end{DoxyCode}
|
|
|