Skip to content

Commit

Permalink
modified call to multiprocessing.Pool;
Browse files Browse the repository at this point in the history
now it creates min(len(policies), cpu_count) processes. This fixes #177 and replaces
#179. In general it does not hurt, previous behavior was to always create cpu_count()

processes, some were probably unused.
  • Loading branch information
enzbus committed Sep 4, 2024
1 parent f45f39c commit 31663bd
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 1 deletion.
5 changes: 4 additions & 1 deletion cvxportfolio/simulator.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import sys
import time
from itertools import starmap
from os import cpu_count
from pathlib import Path

import numpy as np
Expand Down Expand Up @@ -770,7 +771,9 @@ def backtest_many(
if (not parallel) or len(policies) == 1:
result = list(starmap(self._worker, zip_args))
else:
with Pool(initializer=_mp_init, initargs=(Lock(),)) as p:
with Pool(
processes=min(len(policies), cpu_count()),
initializer = _mp_init, initargs = (Lock(),)) as p:
result = p.starmap(self._worker, zip_args)

return list(result)
Expand Down
31 changes: 31 additions & 0 deletions docs/manual.rst
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,7 @@ by providing a single objective and a single list of constraints, and specifying
the ``planning_horizon`` argument to, for example, 2. This simply copies the terms
for each step of planning horizon.

.. _disk-access:

Disk Access
-----------
Expand Down Expand Up @@ -485,6 +486,34 @@ internet access. If you run the test suite (by ``python -m cvxportfolio.tests``)
on a computer without internet access you should see a few tests, mostly
in the ``test_data.py`` module, failing, but most of the test suite will run.

Parallel back-testing
---------------------

You can run multiple back-tests in parallel wit the :meth:`MarketSimulator.backtest_many`
method. It takes a list of policies and returns the corresponding list of
:class:`cvxportfolio.result.BacktestResult`. Also
:meth:`MarketSimulator.optimize_hyperparameters` uses the same approach,
to search over the space of hyper-parameters efficiently.

.. note::

It is not recommended to run multiple Cvxportfolio programs at the same time,
unless you are careful to not :ref:`access on-disk storage <disk-access>`. If
you want to run many back-tests at the same time, you should run a single program
with :meth:`MarketSimulator.backtest_many`.

.. note::

If your Cvxportfolio program uses custom objects, for example
:doc:`a forecaster <examples/user_provided_forecasters>`,
and in that you call complex third party libraries, like machine-learning
ones, parallel back-testing can be problematic. You should in those cases
make sure to :ref:`initialize and finalize <execution-model>` all resources you use.
Cvxportfolio supports the ``multiprocess`` `parallel execution library
<https://multiprocess.readthedocs.io/en/latest/>`_, which may help in such
cases. Simply install ``multiprocess`` in the Python environment to make
Cvxportfolio use it.


CVXPY
-----
Expand Down Expand Up @@ -556,6 +585,8 @@ parallel back-tests, ....
In the next section we explain how Cvxportfolio policy objects work,
which is useful to know when extending them.

.. _execution-model:

Policy execution model
----------------------

Expand Down

0 comments on commit 31663bd

Please sign in to comment.