Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPI warmup auto #2886

Closed
wants to merge 78 commits into from
Closed
Show file tree
Hide file tree
Changes from 75 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
1952caf
Added in switching adaptation
bbbales2 Apr 18, 2019
4dc0fff
Simplified code and added some comments
bbbales2 Apr 20, 2019
8000691
Changed 'switching' to 'auto'
bbbales2 Apr 20, 2019
b1205f3
Added try/catch so that if auto adaptation fails it falls back to dia…
bbbales2 Apr 22, 2019
a6ae242
Merge branch 'develop' of https://github.com/stan-dev/stan into exper…
bbbales2 Sep 4, 2019
baacc34
Fixed bug with regularization of automatically picked metric. Added t…
bbbales2 Sep 11, 2019
76bfd27
first draft of master-slave communication setup for warmup
Oct 30, 2019
7bfe044
unit test for mpi warmup communication
Oct 30, 2019
d57e3d9
inter chain and intra chain MPI communicators
Nov 6, 2019
a98744d
warmup loader MPI test
Nov 14, 2019
8d09106
unit test with mpi gathering of stepsize
Nov 15, 2019
b69492b
update math submodule to 7f4e3a4af5
Dec 18, 2019
1e11650
rhat as convergence critierior for mpi warmup
Jan 8, 2020
bae89e1
campfire warmup with rhat and synced stepsize
Jan 8, 2020
6e1fce8
pass num_chains through run_mpi_adaptive_sampler
Jan 9, 2020
45ed1d4
update submodule
Jan 9, 2020
54f5fa3
cross chain rhat calculation unit test
Jan 10, 2020
ef31851
unit test for cross-chain adapted warmup
Jan 14, 2020
d54c26a
use cross chain warmup for nuts adapted
Jan 14, 2020
1a0f743
stream writer only writes for rank == 0
Jan 15, 2020
84d9807
use degenerated version of compute ess for warmup
Jan 15, 2020
e4c6b50
fix branch of gitmodules
Jan 15, 2020
731b310
rename campfire warmup to cross chain warmup
Jan 15, 2020
b393841
rename campfire warmup to cross chain
Jan 15, 2020
524f750
add mpi cross chain adapter
Jan 22, 2020
c67a5e0
rename mpi adapter; add unit tests
Jan 22, 2020
4547a62
adapt unit/diag nuts inherit cross chain adapter
Jan 22, 2020
c6d98f7
check cross-chain convergence in during sampler transition
Jan 22, 2020
ab8ab7e
1st draft of adapting diag metric
Jan 22, 2020
536d789
use harmonic mean for ESS test
Jan 22, 2020
fc8c3ce
fix broadcasting metrics issue
Jan 22, 2020
a58e2a9
rm old mpi adapt implementation
Jan 22, 2020
7befdde
stream writer only writes to inter chain ranks
Jan 23, 2020
99cc236
rng seed should be from cmdstan
Jan 23, 2020
32f552e
pass `num_cross_chains` to diag_e_adapt function
Jan 24, 2020
16eac20
fix sequential compile failure when no MPI flags issued
Jan 24, 2020
26ca4f3
turn off metric aggregation. update submoduel
Jan 26, 2020
a8da936
unit test for mpi var adaptation
Jan 27, 2020
a703c4d
adapter with logging function and new mpi_var_adaptation
Jan 28, 2020
1a5be75
track all windows for end-of-warmup metric update
Jan 29, 2020
36bd480
use minimum instead of harmonic mean for ESS test
Jan 29, 2020
a34a661
mpi_var_adaptatin under #ifdef STAN_LANG_MPI
Jan 29, 2020
a299c50
clean up cross-chain adapter calls in sampler
Jan 29, 2020
77065b3
pass cross chain window size as function call arg
Jan 30, 2020
1410f2c
add target_ess to arg list
Jan 30, 2020
c35985e
cross_chain_rhat argument
Jan 30, 2020
efe33a5
output min ESS among the chains
Jan 30, 2020
57949e3
mpi_stream_writer
Jan 31, 2020
f53fa46
hack: replace num_warmup in output.csv with cross_chain num_warmup
Feb 4, 2020
8f0f0b1
Revert "hack: replace num_warmup in output.csv with cross_chain num_w…
Feb 4, 2020
ad706bc
write `num_warmup` before `adaption terminated` in csv
Feb 4, 2020
6301595
use Stan's ESS for cross-chain ESS test
Feb 5, 2020
0a1c7ed
no stepsize reset after cross-chain converges
Feb 6, 2020
2c75a62
init commit mpi_warmup_v2
Feb 7, 2020
de0b754
use mpi_cross_chain as single entry to cross-chain methods
Feb 7, 2020
88cfa38
move MPI_ADAPTED_WARMUP macro into adapter
Feb 9, 2020
da2f7b0
simplify cross chain interface
Feb 10, 2020
203e271
mpi warmup for dense_e nuts
Feb 10, 2020
cce0945
tests for mpi_covar_adpt. type traits for cross-chain warmup sampler
Feb 10, 2020
4acb8f7
change sed seed to set id
Feb 11, 2020
2982626
fix: unit nuts doesn't need var; no post warmup when max out
Feb 11, 2020
6a95b53
win loop to decrease counter in croos-chain adapter
Feb 11, 2020
9d5c99a
full campfire version where every window has aggregation
Feb 12, 2020
6da0fa6
harmonic mean for stepsize
Feb 12, 2020
7f6ec86
separate stepsize & metric aggregation
Feb 13, 2020
87356a1
no restart for mpi_var_adapt
Feb 13, 2020
92b8f3f
exclude initial buffer from covar calculation
Feb 14, 2020
83a450a
ignore init buffer draws in mpi var adaptation
Feb 14, 2020
f2835ce
Updated mpi file name adjuster to work with diagnostic files
bbbales2 Feb 15, 2020
9791601
Merge remote-tracking branch 'bbbales2/experimental/warmup-auto' into…
bbbales2 Feb 15, 2020
97af9ac
auto_e metric might be working
bbbales2 Feb 17, 2020
f5c903a
Merge branch 'mpi_warmup_v2' into mpi_warmup_auto
bbbales2 Feb 17, 2020
227a0a6
Updating stan math
bbbales2 Feb 18, 2020
37a3fa1
auto adaptation terminates properly now
bbbales2 Feb 18, 2020
02e93b4
Removed some unused files
bbbales2 Feb 18, 2020
278b1f3
Merge remote-tracking branch 'origin/mpi_warmup_v2' into mpi_warmup_auto
bbbales2 Feb 26, 2020
36af38d
Updated math includes
bbbales2 Feb 28, 2020
a93d90d
Merge remote-tracking branch 'origin/mpi_warmup_v2' into mpi_warmup_auto
bbbales2 Feb 28, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[submodule "lib/stan_math"]
path = lib/stan_math
url = https://github.com/stan-dev/math.git
branch = mpi_warmup_v2
2 changes: 1 addition & 1 deletion lib/stan_math
Submodule stan_math updated 741 files
5 changes: 5 additions & 0 deletions make/mpi_warmup.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ifdef MPI_ADAPTED_WARMUP
CXXFLAGS += -DSTAN_LANG_MPI -DMPI_ADAPTED_WARMUP
CC=mpicxx
CXX=mpicxx
endif
1 change: 1 addition & 0 deletions makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ help:

-include $(HOME)/.config/stan/make.local # user-defined variables
-include make/local # user-defined variables
-include make/mpi_warmup.mk

MATH ?= lib/stan_math/
ifeq ($(OS),Windows_NT)
Expand Down
138 changes: 138 additions & 0 deletions src/stan/callbacks/mpi_stream_writer.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
#ifndef STAN_CALLBACKS_MPI_STREAM_WRITER_HPP
#define STAN_CALLBACKS_MPI_STREAM_WRITER_HPP

#ifdef MPI_ADAPTED_WARMUP

#include <stan/callbacks/writer.hpp>
#include <stan/math/mpi/envionment.hpp>
#include <ostream>
#include <vector>
#include <string>

namespace stan {
namespace callbacks {
/**
* <code>mpi_stream_writer</code> is an implementation
* of <code>writer</code> that writes to a stream.
*/
class mpi_stream_writer : public writer {
public:
/**
* Constructs a stream writer with an output stream
* and an optional prefix for comments.
*
* @param[in, out] output stream to write
* @param[in] comment_prefix string to stream before
* each comment line. Default is "".
*/
mpi_stream_writer(int num_chains, std::ostream& output,
const std::string& comment_prefix = "")
: num_chains_(num_chains), output_(output),
comment_prefix_(comment_prefix)
{}

/**
* Virtual destructor
*/
virtual ~mpi_stream_writer() {}

/**
* Set new value for @c num_chains_.
*
* @param[in] n new value of @c num_chains_
*/
void set_num_chains(int n) {
num_chains_ = n;
}

/**
* Writes a set of names on a single line in csv format followed
* by a newline.
*
* Note: the names are not escaped.
*
* @param[in] names Names in a std::vector
*/
void operator()(const std::vector<std::string>& names) {
write_vector(names);
}

/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] state Values in a std::vector
*/
void operator()(const std::vector<double>& state) {
write_vector(state);
}

/**
* Writes the comment_prefix to the stream followed by a newline.
*/
void operator()() {
if (stan::math::mpi::Session::is_in_inter_chain_comm(num_chains_)) {
output_ << comment_prefix_ << std::endl;
}
}

/**
* Writes the comment_prefix then the message followed by a newline.
*
* @param[in] message A string
*/
void operator()(const std::string& message) {
if (stan::math::mpi::Session::is_in_inter_chain_comm(num_chains_)) {
output_ << comment_prefix_ << message << std::endl;
}
}

private:

/**
* nb. of chains that have its own output stream
*/
int num_chains_;

/**
* Output stream
*/
std::ostream& output_;

/**
* Comment prefix to use when printing comments: strings and blank lines
*/
std::string comment_prefix_;

/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] v Values in a std::vector
*/
template <class T>
void write_vector(const std::vector<T>& v) {
if (stan::math::mpi::Session::is_in_inter_chain_comm(num_chains_)) {
if (v.empty()) return;

typename std::vector<T>::const_iterator last = v.end();
--last;

for (typename std::vector<T>::const_iterator it = v.begin();
it != last; ++it)
output_ << *it << ",";
output_ << v.back() << std::endl;
}
}
};

}
}

#endif

#endif
173 changes: 88 additions & 85 deletions src/stan/callbacks/stream_writer.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,101 +7,104 @@
#include <string>

namespace stan {
namespace callbacks {
namespace callbacks {

/**
* <code>stream_writer</code> is an implementation
* of <code>writer</code> that writes to a stream.
*/
class stream_writer : public writer {
public:
/**
* Constructs a stream writer with an output stream
* and an optional prefix for comments.
*
* @param[in, out] output stream to write
* @param[in] comment_prefix string to stream before
* each comment line. Default is "".
*/
explicit stream_writer(std::ostream& output,
const std::string& comment_prefix = "")
: output_(output), comment_prefix_(comment_prefix) {}
/**
* <code>stream_writer</code> is an implementation
* of <code>writer</code> that writes to a stream.
*/
class stream_writer : public writer {
public:
/**
* Constructs a stream writer with an output stream
* and an optional prefix for comments.
*
* @param[in, out] output stream to write
* @param[in] comment_prefix string to stream before
* each comment line. Default is "".
*/
stream_writer(std::ostream& output,
const std::string& comment_prefix = ""):
output_(output), comment_prefix_(comment_prefix) {}

/**
* Virtual destructor
*/
virtual ~stream_writer() {}
/**
* Virtual destructor
*/
virtual ~stream_writer() {}

/**
* Writes a set of names on a single line in csv format followed
* by a newline.
*
* Note: the names are not escaped.
*
* @param[in] names Names in a std::vector
*/
void operator()(const std::vector<std::string>& names) {
write_vector(names);
}
/**
* Writes a set of names on a single line in csv format followed
* by a newline.
*
* Note: the names are not escaped.
*
* @param[in] names Names in a std::vector
*/
void operator()(const std::vector<std::string>& names) {
write_vector(names);
}

/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] state Values in a std::vector
*/
void operator()(const std::vector<double>& state) { write_vector(state); }
/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] state Values in a std::vector
*/
void operator()(const std::vector<double>& state) {
write_vector(state);
}

/**
* Writes the comment_prefix to the stream followed by a newline.
*/
void operator()() { output_ << comment_prefix_ << std::endl; }
/**
* Writes the comment_prefix to the stream followed by a newline.
*/
void operator()() {
output_ << comment_prefix_ << std::endl;
}

/**
* Writes the comment_prefix then the message followed by a newline.
*
* @param[in] message A string
*/
void operator()(const std::string& message) {
output_ << comment_prefix_ << message << std::endl;
}
/**
* Writes the comment_prefix then the message followed by a newline.
*
* @param[in] message A string
*/
void operator()(const std::string& message) {
output_ << comment_prefix_ << message << std::endl;
}

private:
/**
* Output stream
*/
std::ostream& output_;
private:
/**
* Output stream
*/
std::ostream& output_;

/**
* Comment prefix to use when printing comments: strings and blank lines
*/
std::string comment_prefix_;
/**
* Comment prefix to use when printing comments: strings and blank lines
*/
std::string comment_prefix_;

/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] v Values in a std::vector
*/
template <class T>
void write_vector(const std::vector<T>& v) {
if (v.empty())
return;
/**
* Writes a set of values in csv format followed by a newline.
*
* Note: the precision of the output is determined by the settings
* of the stream on construction.
*
* @param[in] v Values in a std::vector
*/
template <class T>
void write_vector(const std::vector<T>& v) {
if (v.empty()) return;

typename std::vector<T>::const_iterator last = v.end();
--last;
typename std::vector<T>::const_iterator last = v.end();
--last;

for (typename std::vector<T>::const_iterator it = v.begin(); it != last;
++it)
output_ << *it << ",";
output_ << v.back() << std::endl;
}
};
for (typename std::vector<T>::const_iterator it = v.begin();
it != last; ++it)
output_ << *it << ",";
output_ << v.back() << std::endl;
}
};

} // namespace callbacks
} // namespace stan
}
}
#endif
Loading