Skip to content

Commit

Permalink
Merge pull request #331 from WenjieDu/dev
Browse files Browse the repository at this point in the history 
Add Crossformer, Autoformer, PatchTST, DLinear, ETSformer, FEDformer as imputation models
  • Loading branch information
@WenjieDu
WenjieDu authored Apr 1, 2024
2 parents a478836 + 0fbb5ee commit b5d5d1c
Show file tree
Hide file tree
Showing 42 changed files with 4,103 additions and 83 deletions.
18 changes: 15 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ dataset = {"X": X} # X for model input
print(X.shape) # (11988, 48, 37), 11988 samples and each sample has 48 time steps, 37 features

# Model training. This is PyPOTS showtime.
saits = SAITS(n_steps=48, n_features=37, n_layers=2, d_model=256, d_inner=128, n_heads=4, d_k=64, d_v=64, dropout=0.1, epochs=10)
saits = SAITS(n_steps=48, n_features=37, n_layers=2, d_model=256, d_ffn=128, n_heads=4, d_k=64, d_v=64, dropout=0.1, epochs=10)
# Here I use the whole dataset as the training set because ground truth is not visible to the model, you can also split it into train/val/test sets
saits.fit(dataset)
imputation = saits.impute(dataset) # impute the originally-missing values and artificially-missing values
Expand All @@ -197,14 +197,21 @@ This functionality is implemented with the [Microsoft NNI](https://github.com/mi
| **Type** | **Abbr.** | **Full name of the algorithm/model** | **Year** |
| Neural Net | SAITS | Self-Attention-based Imputation for Time Series [^1] | 2023 |
| Neural Net | Transformer | Attention is All you Need [^2];<br>Self-Attention-based Imputation for Time Series [^1];<br><sub>Note: proposed in [^2], and re-implemented as an imputation model in [^1].</sub> | 2017 |
| Neural Net | Crossformer | Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting [^16] | 2023 |
| Neural Net | TimesNet | Temporal 2D-Variation Modeling for General Time Series Analysis [^14] | 2023 |
| Neural Net | Autoformer | Decomposition transformers with auto-correlation for long-term series forecasting [^15] | 2021 |
| Neural Net | PatchTST | A Time Series is Worth 64 Words: Long-Term Forecasting with Transformers [^18] | 2023 |
| Neural Net | DLinear | Are Transformers Effective for Time Series Forecasting? [^17] | 2023 |
| Neural Net | ETSformer | Exponential Smoothing Transformers for Time-series Forecasting [^19] | 2023 |
| Neural Net | FEDformer | Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting [^20] | 2022 |
| Neural Net | Autoformer | Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting [^15] | 2021 |
| Neural Net | CSDI | Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation [^12] | 2021 |
| Neural Net | US-GAN | Unsupervised GAN for Multivariate Time Series Imputation [^10] | 2021 |
| Neural Net | GP-VAE | Gaussian Process Variational Autoencoder [^11] | 2020 |
| Neural Net | BRITS | Bidirectional Recurrent Imputation for Time Series [^3] | 2018 |
| Neural Net | M-RNN | Multi-directional Recurrent Neural Network [^9] | 2019 |
| Naive | LOCF | Last Observation Carried Forward | - |
| Naive | LOCF/NOCB | Last Observation Carried Forward / Next Observation Carried Backward | - |
| Naive | Median | Median Value Imputation | - |
| Naive | Mean | Mean Value Imputation | - |
| ***`Classification`*** | 🚥 | 🚥 | 🚥 |
| **Type** | **Abbr.** | **Full name of the algorithm/model/paper** | **Year** |
| Neural Net | BRITS | Bidirectional Recurrent Imputation for Time Series [^3] | 2018 |
Expand Down Expand Up @@ -320,6 +327,11 @@ PyPOTS community is open, transparent, and surely friendly. Let's work together
[^13]: Rubin, D. B. (1976). [Inference and missing data](https://academic.oup.com/biomet/article-abstract/63/3/581/270932). *Biometrika*.
[^14]: Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., & Long, M. (2023). [TimesNet: Temporal 2d-variation modeling for general time series analysis](https://openreview.net/forum?id=ju_Uqw384Oq). *ICLR 2023*
[^15]: Wu, H., Xu, J., Wang, J., & Long, M. (2021). [Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting](https://proceedings.neurips.cc/paper/2021/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html). *NeurIPS 2021*.
[^16]: Zhang, Y., & Yan, J. (2023). [Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting](https://openreview.net/forum?id=vSVLM2j9eie). *ICLR 2023*.
[^17]: Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). [Are transformers effective for time series forecasting?](https://ojs.aaai.org/index.php/AAAI/article/view/26317). *AAAI 2023*
[^18]: Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2023). [A time series is worth 64 words: Long-term forecasting with transformers](https://openreview.net/forum?id=Jbdc0vTOcol). *ICLR 2023*
[^19]: Woo, G., Liu, C., Sahoo, D., Kumar, A., & Hoi, S. (2023). [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://openreview.net/forum?id=5m_3whfo483). *ICLR 2023*
[^20]: Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022). [FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting](https://proceedings.mlr.press/v162/zhou22g.html). *ICML 2022*.


<details>
Expand Down
45 changes: 27 additions & 18 deletions docs/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,24 +170,33 @@ PyPOTS supports imputation, classification, clustering, and forecasting tasks on
🌟 Since **v0.2**, all neural-network models in PyPOTS has got hyperparameter-optimization support.
This functionality is implemented with the `Microsoft NNI <https://github.com/microsoft/nni>`_ framework.

============================== ================ ======================================================================================== ====== =========
Task Type Algorithm Year Reference
============================== ================ ======================================================================================== ====== =========
Imputation Neural Net SAITS (Self-Attention-based Imputation for Time Series) 2023 :cite:`du2023SAITS`
Imputation Neural Net Transformer 2017 :cite:`vaswani2017Transformer`, :cite:`du2023SAITS`
Imputation Neural Net TimesNet 2023 :cite:`wu2023timesnet`
Imputation Neural Net US-GAN (Unsupervised GAN for Multivariate Time Series Imputation) 2021 :cite:`miao2021SSGAN`
Imputation Neural Net CSDI (Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation) 2021 :cite:`tashiro2021csdi`
Imputation Neural Net GP-VAE (Gaussian Process Variational Autoencoder) 2020 :cite:`fortuin2020GPVAEDeep`
Imputation, Classification Neural Net BRITS (Bidirectional Recurrent Imputation for Time Series) 2018 :cite:`cao2018BRITS`
Imputation Neural Net M-RNN (Multi-directional Recurrent Neural Network) 2019 :cite:`yoon2019MRNN`
Imputation Naive LOCF (Last Observation Carried Forward) / /
Classification Neural Net GRU-D 2018 :cite:`che2018GRUD`
Classification Neural Net Raindrop 2022 :cite:`zhang2022Raindrop`
Clustering Neural Net CRLI (Clustering Representation Learning on Incomplete time-series data) 2021 :cite:`ma2021CRLI`
Clustering Neural Net VaDER (Variational Deep Embedding with Recurrence) 2019 :cite:`dejong2019VaDER`
Forecasting Probabilistic BTTF (Bayesian Temporal Tensor Factorization) 2021 :cite:`chen2021BTMF`
============================== ================ ======================================================================================== ====== =========
============================== ================ ========================================================================================================= ====== =========
Task Type Algorithm Year Reference
============================== ================ ========================================================================================================= ====== =========
Imputation Neural Net SAITS (Self-Attention-based Imputation for Time Series) 2023 :cite:`du2023SAITS`
Imputation Neural Net Transformer 2017 :cite:`vaswani2017Transformer`, :cite:`du2023SAITS`
Imputation Neural Net Crossformer (Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting) 2023 :cite:`nie2023patchtst`
Imputation Neural Net TimesNet (Temporal 2D-Variation Modeling for General Time Series Analysis) 2023 :cite:`wu2023timesnet`
Imputation Neural Net PatchTST (A Time Series is Worth 64 Words: Long-Term Forecasting with Transformers) 2023 :cite:`nie2023patchtst`
Imputation Neural Net DLinear (Are transformers effective for time series forecasting?) 2023 :cite:`zeng2023dlinear`
Imputation Neural Net ETSformer (Exponential Smoothing Transformers for Time-series Forecasting) 2023 :cite:`woo2023etsformer`
Imputation Neural Net FEDformer (Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting) 2022 :cite:`zhou2022fedformer`
Imputation Neural Net Autoformer (Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting) 2021 :cite:`wu2021autoformer`
Imputation Neural Net US-GAN (Unsupervised GAN for Multivariate Time Series Imputation) 2021 :cite:`miao2021SSGAN`
Imputation Neural Net CSDI (Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation) 2021 :cite:`tashiro2021csdi`
Imputation Neural Net GP-VAE (Gaussian Process Variational Autoencoder) 2020 :cite:`fortuin2020GPVAEDeep`
Imputation, Classification Neural Net BRITS (Bidirectional Recurrent Imputation for Time Series) 2018 :cite:`cao2018BRITS`
Imputation Neural Net M-RNN (Multi-directional Recurrent Neural Network) 2019 :cite:`yoon2019MRNN`
Imputation Naive LOCF (Last Observation Carried Forward) / /
Imputation Naive NOCB (Next Observation Carried Backward) / /
Imputation Naive Median (Median Value Imputation) / /
Imputation Naive Mean (Mean Value Imputation) / /
Classification Neural Net GRU-D 2018 :cite:`che2018GRUD`
Classification Neural Net Raindrop 2022 :cite:`zhang2022Raindrop`
Clustering Neural Net CRLI (Clustering Representation Learning on Incomplete time-series data) 2021 :cite:`ma2021CRLI`
Clustering Neural Net VaDER (Variational Deep Embedding with Recurrence) 2019 :cite:`dejong2019VaDER`
Forecasting Probabilistic BTTF (Bayesian Temporal Tensor Factorization) 2021 :cite:`chen2021BTMF`
============================== ================ ========================================================================================================= ====== =========


❖ Citing PyPOTS
Expand Down
45 changes: 45 additions & 0 deletions docs/pypots.imputation.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@ pypots.imputation.transformer
:show-inheritance:
:inherited-members:

pypots.imputation.crossformer
------------------------------

.. automodule:: pypots.imputation.crossformer
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.timesnet
------------------------------

Expand All @@ -28,6 +37,42 @@ pypots.imputation.timesnet
:show-inheritance:
:inherited-members:

pypots.imputation.patchtst
------------------------------

.. automodule:: pypots.imputation.patchtst
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.dlinear
------------------------------

.. automodule:: pypots.imputation.dlinear
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.etsformer
------------------------------

.. automodule:: pypots.imputation.etsformer
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.fedformer
------------------------------

.. automodule:: pypots.imputation.fedformer
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.autoformer
------------------------------

Expand Down
52 changes: 52 additions & 0 deletions docs/references.bib
View file
Original file line number Diff line number Diff line change
Expand Up @@ -484,3 +484,55 @@ @inproceedings{wu2021autoformer
volume = {34},
year = {2021}
}

@inproceedings{zhang2023crossformer,
title={Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting},
author={Yunhao Zhang and Junchi Yan},
booktitle={The Eleventh International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=vSVLM2j9eie}
}

@inproceedings{zeng2023dlinear,
title={Are Transformers Effective for Time Series Forecasting?},
volume={37},
url={https://ojs.aaai.org/index.php/AAAI/article/view/26317},
DOI={10.1609/aaai.v37i9.26317},
number={9},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Zeng, Ailing and Chen, Muxi and Zhang, Lei and Xu, Qiang},
year={2023},
month={Jun.},
pages={11121-11128}
}

@inproceedings{nie2023patchtst,
title={A Time Series is Worth 64 Words: Long-term Forecasting with Transformers},
author={Yuqi Nie and Nam H Nguyen and Phanwadee Sinthong and Jayant Kalagnanam},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=Jbdc0vTOcol}
}

@inproceedings{woo2023etsformer,
title={{ETS}former: Exponential Smoothing Transformers for Time-series Forecasting},
author={Gerald Woo and Chenghao Liu and Doyen Sahoo and Akshat Kumar and Steven Hoi},
booktitle={The Eleventh International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=5m_3whfo483}
}

@inproceedings{zhou2022fedformer,
title = {{FED}former: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting},
author = {Zhou, Tian and Ma, Ziqing and Wen, Qingsong and Wang, Xue and Sun, Liang and Jin, Rong},
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
pages = {27268--27286},
year = {2022},
editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
volume = {162},
series = {Proceedings of Machine Learning Research},
month = {17--23 Jul},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v162/zhou22g/zhou22g.pdf},
url = {https://proceedings.mlr.press/v162/zhou22g.html},
}
4 changes: 3 additions & 1 deletion environment-dev.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ dependencies:
- conda-forge::h5py
- conda-forge::numpy
- conda-forge::scipy
- conda-forge::sympy
- conda-forge::python
- conda-forge::einops
- conda-forge::pandas
- conda-forge::matplotlib
- conda-forge::tensorboard
Expand All @@ -25,8 +27,8 @@ dependencies:

# optional
- pyg::pyg
- pyg::pytorch-scatter
- pyg::pytorch-sparse
- pyg::pytorch-scatter

# test
- conda-forge::pytest-cov
Expand Down
Loading

0 comments on commit b5d5d1c

Please sign in to comment.