diff --git a/README.md b/README.md index 113990c..d2a2ee7 100644 --- a/README.md +++ b/README.md @@ -31,19 +31,26 @@ print(log_prob.shape) # (100,) print(x_new.shape) # (50, 3) ``` -We provide more examples [here](examples/README.md). +We provide more examples [here](examples/). ## Installing -We support Python versions 3.7 and upwards. +Install the package: -Install via git: -```python +``` +pip install git+https://github.com/davidnabergoj/normalizing-flows.git +``` + +Setup for development: + +``` git clone https://github.com/davidnabergoj/normalizing-flows.git cd normalizing-flows pip install -r requirements.txt ``` +We support Python versions 3.7 and upwards. + ## Brief background A normalizing flow (NF) is a flexible distribution, defined as a bijective transformation of a simple statistical diff --git a/examples/Computing log determinants.md b/examples/Computing log determinants.md new file mode 100644 index 0000000..17e236c --- /dev/null +++ b/examples/Computing log determinants.md @@ -0,0 +1,24 @@ +# Computing the log determinant of the Jacobian + +We show how to compute and retrieve the log determinant of the Jacobian of a bijective transformation. +We use Real NVP as an example, but you can replace it with any other bijection from `normalizing_flows.bijections`. +The code is as follows: + +```python +import torch +from normalizing_flows import Flow +from normalizing_flows.bijections import RealNVP + +torch.manual_seed(0) + +batch_shape = (5, 7) +event_shape = (2, 3) +x = torch.randn(size=(*batch_shape, *event_shape)) +z = torch.randn(size=(*batch_shape, *event_shape)) + +bijection = RealNVP(event_shape=event_shape) +flow = Flow(bijection) + +_, log_det_forward = flow.bijection.forward(x) +_, log_det_inverse = flow.bijection.inverse(z) +``` \ No newline at end of file diff --git a/examples/Modifying architectures.md b/examples/Modifying architectures.md new file mode 100644 index 0000000..aec3d68 --- /dev/null +++ b/examples/Modifying architectures.md @@ -0,0 +1,33 @@ +# Creating and modifying bijection architectures + +We give an example on how to modify a bijection's architecture. +We use the Masked Autoregressive Flow (MAF) as an example. +We can manually set the number of invertible layers as follows: +```python +from normalizing_flows.bijections import MAF + +event_shape = (10,) +flow = MAF(event_shape=event_shape, n_layers=5) +``` + +For specific changes, we can create individual invertible layers and combine them into a bijection. +MAF uses affine masked autoregressive layers with permutations in between. +We can import these layers set their parameters as desired. +For example, to change the number of layers in the MAF conditioner and its hidden layer sizes, we proceed as follows: +```python +from normalizing_flows.bijections import BijectiveComposition +from normalizing_flows.bijections.finite.autoregressive.layers import AffineForwardMaskedAutoregressive +from normalizing_flows.bijections.finite.linear import ReversePermutation + +event_shape = (10,) +flow = BijectiveComposition( + event_shape=event_shape, + layers=[ + AffineForwardMaskedAutoregressive(event_shape=event_shape, n_layers=4, n_hidden=20), + ReversePermutation(event_shape=event_shape), + AffineForwardMaskedAutoregressive(event_shape=event_shape, n_layers=3, n_hidden=7), + ReversePermutation(event_shape=event_shape), + AffineForwardMaskedAutoregressive(event_shape=event_shape, n_layers=5, n_hidden=13) + ] +) +``` \ No newline at end of file diff --git a/examples/README.md b/examples/README.md deleted file mode 100644 index 9cadc02..0000000 --- a/examples/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Examples - -We provide minimal working examples on how to perform various common tasks with normalizing flows. -We use Real NVP as an example, but you can replace it with any other bijection from `normalizing_flows.bijections`. - -## Training a normalizing flow on a fixed dataset -```python -import torch -from normalizing_flows import Flow -from normalizing_flows.bijections import RealNVP - -torch.manual_seed(0) - -# We support arbitrary event and batch shapes -event_shape = (2, 3) -batch_shape = (5, 7) -x_train = torch.randn(size=(*batch_shape, *event_shape)) - -bijection = RealNVP(event_shape=event_shape) -flow = Flow(bijection) - -flow.fit(x_train, show_progress=True) -``` - -## Computing the log determinant of the Jacobian transformation given a Flow -```python -import torch -from normalizing_flows import Flow -from normalizing_flows.bijections import RealNVP - -torch.manual_seed(0) - -batch_shape = (5, 7) -event_shape = (2, 3) -x = torch.randn(size=(*batch_shape, *event_shape)) -z = torch.randn(size=(*batch_shape, *event_shape)) - -bijection = RealNVP(event_shape=event_shape) -flow = Flow(bijection) - -_, log_det_forward = flow.bijection.forward(x) -# log_det_forward.shape == batch_shape - -_, log_det_inverse = flow.bijection.inverse(z) -# log_det_inverse.shape == batch_shape -``` \ No newline at end of file diff --git a/examples/Training a normalizing flow.md b/examples/Training a normalizing flow.md new file mode 100644 index 0000000..1078bfc --- /dev/null +++ b/examples/Training a normalizing flow.md @@ -0,0 +1,29 @@ +# Training normalizing flow models on a dataset + +We demonstrate how to train a normalizing flow on a dataset. +We use Real NVP as an example, but you can replace it with any other bijection from `normalizing_flows.bijections`. +The code is as follows: + +```python +import torch +from normalizing_flows import Flow +from normalizing_flows.bijections import RealNVP + +torch.manual_seed(0) + +# We support arbitrary event and batch shapes +event_shape = (2, 3) +batch_shape = (5, 7) +x_train = torch.randn(size=(*batch_shape, *event_shape)) + +bijection = RealNVP(event_shape=event_shape) +flow = Flow(bijection) + +flow.fit(x_train, show_progress=True) +``` + +To modify the learning rate, simply use the `lr` keyword argument in `flow.fit(...)`: + +```python +flow.fit(x_train, show_progress=True, lr=0.001) +``` \ No newline at end of file diff --git a/normalizing_flows/bijections/continuous/__init__.py b/normalizing_flows/bijections/continuous/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/__init__.py b/normalizing_flows/bijections/finite/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/autoregressive/__init__.py b/normalizing_flows/bijections/finite/autoregressive/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/autoregressive/conditioners/__init__.py b/normalizing_flows/bijections/finite/autoregressive/conditioners/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/autoregressive/transformers/__init__.py b/normalizing_flows/bijections/finite/autoregressive/transformers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/autoregressive/transformers/combination/base.py b/normalizing_flows/bijections/finite/autoregressive/transformers/combination/base.py index b08bb2b..fb431ed 100644 --- a/normalizing_flows/bijections/finite/autoregressive/transformers/combination/base.py +++ b/normalizing_flows/bijections/finite/autoregressive/transformers/combination/base.py @@ -1,12 +1,12 @@ import torch -from typing import Tuple +from typing import Tuple, List from normalizing_flows.bijections.finite.autoregressive.transformers.base import Transformer from normalizing_flows.utils import get_batch_shape class Combination(Transformer): - def __init__(self, event_shape: torch.Size, components: list[Transformer]): + def __init__(self, event_shape: torch.Size, components: List[Transformer]): super().__init__(event_shape) self.components = components self.n_components = len(self.components) diff --git a/normalizing_flows/bijections/finite/autoregressive/transformers/spline/__init__.py b/normalizing_flows/bijections/finite/autoregressive/transformers/spline/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/normalizing_flows/bijections/finite/residual/__init__.py b/normalizing_flows/bijections/finite/residual/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..38b639f --- /dev/null +++ b/setup.py @@ -0,0 +1,48 @@ +from setuptools import setup, find_packages +import pathlib + +here = pathlib.Path(__file__).parent.resolve() + +long_description = (here / "README.md").read_text(encoding="utf-8") + +setup( + name="normalizing-flows", + version="0.1", + description="Modern normalizing flows in Python", + long_description=long_description, + long_description_content_type="text/markdown", + url="https://github.com/davidnabergoj/normalizing-flows", + author="David Nabergoj", + author_email="david.nabergoj@fri.uni-lj.si", + classifiers=[ # Optional + "Development Status :: 3 - Alpha", + "Intended Audience :: Science/Research", + "Topic :: Software Development :: Build Tools", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3 :: Only", + ], + keywords=[ + "python", + "machine-learning", + "pytorch", + "generative-model", + "sampling", + "density-estimation", + "normalizing-flow" + ], + packages=find_packages(exclude=["test"]), + python_requires=">=3.7, <4", + install_requires=[ + "torch>=2.0.1", + "numpy", + "torchdiffeq", + "tqdm" + ], + project_urls={ + "Bug Reports": "https://github.com/davidnabergoj/normalizing-flows/issues", + }, +)