Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bridgestan failure with Hessian AD enabled #237

Closed
sethaxen opened this issue Jun 24, 2024 · 11 comments · Fixed by stan-dev/stan#3294
Closed

Bridgestan failure with Hessian AD enabled #237

sethaxen opened this issue Jun 24, 2024 · 11 comments · Fixed by stan-dev/stan#3294

Comments

@sethaxen
Copy link
Contributor

I'm not certain if this is an issue with bridgestan or coming from somewhere else in the toolchain, but if I install gxx from conda-forge, install bridgestan from pypi, and then compile a model requesting BRIDGESTAN_AD_HESSIAN=true, then it errors if I have an expression like target += y for a vector y. It doesn't fail if I use target += sum(y), and it also doesn't fail if I don't set BRIDGESTAN_AD_HESSIAN=true.

Here's a reproducible example:
test.stan:

data {
  int<lower=1> N;
}
parameters {
  vector[N] y;
}
model {
  target += y;    
}

Script:

conda create -n bridgestan-test -c conda-forge python=3.11 gxx=13.2.0 pip
conda activate bridgestan-test
python -m pip install bridgestan==2.5.0
python -c '
import bridgestan as bs
stan_file = "./test.stan"
bs.compile_model(stan_file, make_args=["BRIDGESTAN_AD_HESSIAN=true"])
'

Error:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/python3.11/site-packages/bridgestan/compile.py", line 142, in compile_model
    raise RuntimeError(error)
RuntimeError: Command make BRIDGESTAN_AD_HESSIAN=true STANCFLAGS=--include-paths=. /home/sethaxen/bridgestan_test/test_model.so failed with code 2.
stdout:

--- Translating Stan model to C++ code ---
./bin/stanc --include-paths=. --o=/home/sethaxen/bridgestan_test/test.hpp /home/sethaxen/bridgestan_test/test.stan

--- Compiling C++ code ---
g++ -std=c++17 -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -Wno-class-memaccess      -I ./stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I ./stan/src -I ./stan/lib/rapidjson_1.1.0/ -I ./stan/lib/stan_math/ -I ./stan/lib/stan_math/lib/eigen_3.4.0 -I ./stan/lib/stan_math/lib/boost_1.84.0 -I ./stan/lib/stan_math/lib/sundials_6.1.1/include -I ./stan/lib/stan_math/lib/sundials_6.1.1/src/sundials -fPIC -fvisibility=hidden -fvisibility-inlines-hidden    -DBOOST_DISABLE_ASSERTS         -DBRIDGESTAN_EXPORT -DSTAN_MODEL_FVAR_VAR -DBRIDGESTAN_AD_HESSIAN  -c  -x c++ -o /home/sethaxen/bridgestan_test/test.o /home/sethaxen/bridgestan_test/test.hpp
rm /home/sethaxen/bridgestan_test/test.hpp

stderr:
In file included from ./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/traits.hpp:21,
                 from ./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/storage.hpp:27,
                 from ./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/vector.hpp:21,
                 from ./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/odeint/util/ublas_wrapper.hpp:23,
                 from ./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/odeint.hpp:25,
                 from ./stan/lib/stan_math/stan/math/prim/functor/ode_rk45.hpp:9,
                 from ./stan/lib/stan_math/stan/math/prim/functor/integrate_ode_rk45.hpp:6,
                 from ./stan/lib/stan_math/stan/math/prim/functor.hpp:16,
                 from ./stan/lib/stan_math/stan/math/rev/fun.hpp:204,
                 from ./stan/lib/stan_math/stan/math/rev.hpp:12,
                 from ./stan/lib/stan_math/stan/math.hpp:19,
                 from ./stan/src/stan/model/model_header.hpp:4,
                 from /home/sethaxen/bridgestan_test/test.hpp:2:
./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/detail/iterator.hpp:111:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  111 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
In file included from /home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_algobase.h:65,
                 from /home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/specfun.h:43,
                 from /home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/cmath:3699,
                 from ./stan/lib/stan_math/lib/eigen_3.4.0/Eigen/src/Core/util/Macros.h:679,
                 from ./stan/lib/stan_math/lib/eigen_3.4.0/Eigen/Core:19,
                 from ./stan/lib/stan_math/lib/eigen_3.4.0/Eigen/Dense:1,
                 from ./stan/lib/stan_math/stan/math/prim/fun/Eigen.hpp:22,
                 from ./stan/lib/stan_math/stan/math/rev.hpp:4:
/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/detail/iterator.hpp:149:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  149 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
./stan/lib/stan_math/lib/boost_1.84.0/boost/numeric/ublas/detail/iterator.hpp:204:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  204 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
In file included from ./stan/lib/stan_math/stan/math/rev/fun/accumulator.hpp:7,
                 from ./stan/lib/stan_math/stan/math/rev/fun.hpp:10:
./stan/lib/stan_math/stan/math/prim/fun/accumulator.hpp: In instantiation of 'void stan::math::accumulator<T, <template-parameter-1-2> >::add(const S&) [with S = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; stan::require_matrix_t<S>* <anonymous> = 0; T = stan::math::fvar<stan::math::var_value<double> >; <template-parameter-1-2> = void]':
/home/sethaxen/bridgestan_test/test.hpp:88:0:   required from 'stan::scalar_type_t<T2> test_model_namespace::test_model::log_prob_impl(VecR&, VecI&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; VecR = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; VecI = Eigen::Matrix<int, -1, 1>; stan::require_vector_like_t<VecR>* <anonymous> = 0; stan::require_vector_like_vt<std::is_integral, VecI>* <anonymous> = 0; stan::require_not_st_var<VecR>* <anonymous> = 0; stan::scalar_type_t<T2> = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
/home/sethaxen/bridgestan_test/test.hpp:332:0:   required from 'T_ test_model_namespace::test_model::log_prob(Eigen::Matrix<T_a, -1, 1>&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; T_ = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:241:0:   required from 'stan::math::fvar<stan::math::var_value<double> > stan::model::model_base_crtp<M>::log_prob(Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>&, std::ostream*) const [with M = test_model_namespace::test_model; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:238:0:   required from here
./stan/lib/stan_math/stan/math/prim/fun/accumulator.hpp:53:35: error: no matching function for call to 'sum(const Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>&)'
   53 |     buf_.push_back(stan::math::sum(m));
      |                    ~~~~~~~~~~~~~~~^~~
In file included from ./stan/lib/stan_math/stan/math/prim/meta/possibly_sum.hpp:4,
                 from ./stan/lib/stan_math/stan/math/prim/meta.hpp:119,
                 from ./stan/lib/stan_math/stan/math/rev/core/accumulate_adjoints.hpp:4,
                 from ./stan/lib/stan_math/stan/math/rev/core.hpp:4,
                 from ./stan/lib/stan_math/stan/math/rev.hpp:10:
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:21:10: note: candidate: 'template<class T, stan::require_stan_scalar_t<T>* <anonymous> > T stan::math::sum(T&&)'
   21 | inline T sum(T&& m) {
      |          ^~~
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:21:10: note:   template argument deduction/substitution failed:
In file included from /home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_pair.h:60,
                 from /home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/bits/stl_algobase.h:64:
/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/type_traits: In substitution of 'template<bool _Cond, class _Tp> using std::enable_if_t = typename std::enable_if::type [with bool _Cond = false; _Tp = void]':
./stan/lib/stan_math/stan/math/prim/meta/require_helpers.hpp:19:7:   required by substitution of 'template<class Check> using stan::require_t = std::enable_if_t<Check::value> [with Check = stan::is_stan_scalar<Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1> >]'
./stan/lib/stan_math/stan/math/prim/meta/is_stan_scalar.hpp:39:7:   required by substitution of 'template<class T> using stan::require_stan_scalar_t = stan::require_t<stan::is_stan_scalar<typename std::decay<_Tp>::type> > [with T = const Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>&]'
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:20:51:   required from 'void stan::math::accumulator<T, <template-parameter-1-2> >::add(const S&) [with S = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; stan::require_matrix_t<S>* <anonymous> = 0; T = stan::math::fvar<stan::math::var_value<double> >; <template-parameter-1-2> = void]'
/home/sethaxen/bridgestan_test/test.hpp:88:0:   required from 'stan::scalar_type_t<T2> test_model_namespace::test_model::log_prob_impl(VecR&, VecI&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; VecR = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; VecI = Eigen::Matrix<int, -1, 1>; stan::require_vector_like_t<VecR>* <anonymous> = 0; stan::require_vector_like_vt<std::is_integral, VecI>* <anonymous> = 0; stan::require_not_st_var<VecR>* <anonymous> = 0; stan::scalar_type_t<T2> = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
/home/sethaxen/bridgestan_test/test.hpp:332:0:   required from 'T_ test_model_namespace::test_model::log_prob(Eigen::Matrix<T_a, -1, 1>&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; T_ = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:241:0:   required from 'stan::math::fvar<stan::math::var_value<double> > stan::model::model_base_crtp<M>::log_prob(Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>&, std::ostream*) const [with M = test_model_namespace::test_model; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:238:0:   required from here
/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/gcc/x86_64-conda-linux-gnu/13.2.0/include/c++/type_traits:2610:11: error: no type named 'type' in 'struct std::enable_if<false, void>'
 2610 |     using enable_if_t = typename enable_if<_Cond, _Tp>::type;
      |           ^~~~~~~~~~~
./stan/lib/stan_math/stan/math/prim/fun/accumulator.hpp: In instantiation of 'void stan::math::accumulator<T, <template-parameter-1-2> >::add(const S&) [with S = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; stan::require_matrix_t<S>* <anonymous> = 0; T = stan::math::fvar<stan::math::var_value<double> >; <template-parameter-1-2> = void]':
/home/sethaxen/bridgestan_test/test.hpp:88:0:   required from 'stan::scalar_type_t<T2> test_model_namespace::test_model::log_prob_impl(VecR&, VecI&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; VecR = Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>; VecI = Eigen::Matrix<int, -1, 1>; stan::require_vector_like_t<VecR>* <anonymous> = 0; stan::require_vector_like_vt<std::is_integral, VecI>* <anonymous> = 0; stan::require_not_st_var<VecR>* <anonymous> = 0; stan::scalar_type_t<T2> = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
/home/sethaxen/bridgestan_test/test.hpp:332:0:   required from 'T_ test_model_namespace::test_model::log_prob(Eigen::Matrix<T_a, -1, 1>&, std::ostream*) const [with bool propto__ = false; bool jacobian__ = false; T_ = stan::math::fvar<stan::math::var_value<double> >; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:241:0:   required from 'stan::math::fvar<stan::math::var_value<double> > stan::model::model_base_crtp<M>::log_prob(Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>&, std::ostream*) const [with M = test_model_namespace::test_model; std::ostream = std::basic_ostream<char>]'
./stan/src/stan/model/model_base_crtp.hpp:238:0:   required from here
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:33:10: note: candidate: 'template<class T, stan::require_not_var_t<T>* <anonymous> > T stan::math::sum(const std::vector<T1>&)'
   33 | inline T sum(const std::vector<T>& m) {
      |          ^~~
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:33:10: note:   template argument deduction/substitution failed:
./stan/lib/stan_math/stan/math/prim/fun/accumulator.hpp:53:35: note:   'const Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>' is not derived from 'const std::vector<T1>'
   53 |     buf_.push_back(stan::math::sum(m));
      |                    ~~~~~~~~~~~~~~~^~~
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:46:24: note: candidate: 'template<class T, stan::require_eigen_vt<std::is_arithmetic, T>* <anonymous> > stan::value_type_t<T> stan::math::sum(const T&)'
   46 | inline value_type_t<T> sum(const T& m) {
      |                        ^~~
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:46:24: note:   template argument deduction/substitution failed:
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:59:24: note: candidate: 'template<class T, stan::require_eigen_vt<stan::is_complex, T>* <anonymous> > stan::value_type_t<T> stan::math::sum(const T&)'
   59 | inline value_type_t<T> sum(const T& m) {
      |                        ^~~
./stan/lib/stan_math/stan/math/prim/fun/sum.hpp:59:24: note:   template argument deduction/substitution failed:
In file included from ./stan/lib/stan_math/stan/math/rev/fun/accumulator.hpp:6:
./stan/lib/stan_math/stan/math/rev/fun/sum.hpp:26:12: note: candidate: 'template<class Alloc> stan::math::var stan::math::sum(const std::vector<var_value<double>, Alloc>&)'
   26 | inline var sum(const std::vector<var, Alloc>& m) {
      |            ^~~
./stan/lib/stan_math/stan/math/rev/fun/sum.hpp:26:12: note:   template argument deduction/substitution failed:
./stan/lib/stan_math/stan/math/prim/fun/accumulator.hpp:53:35: note:   'const Eigen::Matrix<stan::math::fvar<stan::math::var_value<double> >, -1, 1>' is not derived from 'const std::vector<stan::math::var_value<double>, Alloc>'
   53 |     buf_.push_back(stan::math::sum(m));
      |                    ~~~~~~~~~~~~~~~^~~
./stan/lib/stan_math/stan/math/rev/fun/sum.hpp:47:12: note: candidate: 'template<class T, stan::require_rev_matrix_t<T>* <anonymous> > stan::math::var stan::math::sum(const T&)'
   47 | inline var sum(const T& x) {
      |            ^~~
./stan/lib/stan_math/stan/math/rev/fun/sum.hpp:47:12: note:   template argument deduction/substitution failed:
make: *** [Makefile:78: /home/sethaxen/bridgestan_test/test.o] Error 1
@WardBrian
Copy link
Collaborator

I can reproduce the failure by compiling a cmdstan model with CXXFLAGS+=-DSTAN_MODEL_FVAR_VAR (which is what BridgeStan is actually defining), so it seems like this is a Stan math library issue. Digging into it now.

@WardBrian
Copy link
Collaborator

stan-dev/math#2535 seems related

@WardBrian
Copy link
Collaborator

It appears this was an include-order issue in stan-dev/stan

@sethaxen
Copy link
Contributor Author

Thanks, @WardBrian for the diagnosis and the fix! In the meantime, is there a workaround? e.g. if I replace target += y with target += sum(y), will I still hit this issue? And are there other Stan function calls this would cause to error?

@WardBrian
Copy link
Collaborator

It’s hard to say what other functions would be affected - luckily, they would all manifest in compiler errors, so if they compile you can assume it didn’t run into this problem.

If you run into others, a valid workaround for now is probably just applying the patch I submitted to Stan to your local bridgestan copy - it’s only a few lines changed

@sethaxen
Copy link
Contributor Author

Hi @WardBrian, I hit a perhaps unrelated issue that only shows up when compiling for Hessian AD if the target is incremented by a constant in the model block.:

test.stan

data {
  int<lower=1> N;
}
parameters {
  vector[N] y;
}
model {
  target += 0;
}

Running the same command as in #237 (comment) we get the error:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/home/sethaxen/software/mambaforge/envs/bridgestan-test/lib/python3.11/site-packages/bridgestan/compile.py", line 142, in compile_model
    raise RuntimeError(error)
RuntimeError: Command make BRIDGESTAN_AD_HESSIAN=true STANCFLAGS=--include-paths=. /home/sethaxen/bridgestan_test/test_model.so failed with code 2.
stdout:

--- Translating Stan model to C++ code ---
./bin/stanc --include-paths=. --o=/home/sethaxen/bridgestan_test/test.hpp /home/sethaxen/bridgestan_test/test.stan

stderr:
./bin/stanc: line 1: Not: command not found
make: *** [Makefile:73: /home/sethaxen/bridgestan_test/test.hpp] Error 127

If it's incremented by a non-constant in an _lp function we still get this error, e.g.
test.stan:

functions {
  vector foo_lp(vector y) {
    target += sum(y);
    return y;
  }
}
data {
  int<lower=1> N;
}
parameters {
  vector[N] y;
}
transformed parameters {
  vector[N] x = foo_lp(y);
}
model {
  target += 1;
}

This probably also is not a bridgestan issue, but I'm not sure where to report it.

@WardBrian
Copy link
Collaborator

./bin/stanc: line 1: Not: command not found

This is odd -- can you check in the bridgestan folder if stanc does exist in bin/ ?

@sethaxen
Copy link
Contributor Author

This is odd -- can you check in the bridgestan folder if stanc does exist in bin/ ?

Yes, it's there. If I just increment by target += 0 * sum(y) then compilation was successful.

Weirdly, I deleted .bridgestan and reinstalled everything, and now this issue is mysteriously resolved. But it wasn't just on my machine; I've been seeing this on CI as well but for different models. (e.g. https://github.com/bob-carpenter/transforms/actions/runs/9713116976/job/26809228120#step:7:264). But anyways, seems unrelated to this and probably something to do with steps I took to install bridgestan.

@WardBrian
Copy link
Collaborator

Since stanc is downloaded from github on first use, maybe something was rate-limiting/blocking you, or perhaps github's servers hiccuped?

@sethaxen
Copy link
Contributor Author

Perhaps, though it still seems to be an issue. Is there a way to manually trigger the download of stanc other than compiling a dummy model? I'm wondering if there's somehow an issue with doing this while running pytest.

@WardBrian
Copy link
Collaborator

make bin/stanc (bin/stanc.exe on Windows) should do it. But we use pytest ourselves for the Python bindings and this doesn't seem to be an issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants