From 0128ddaf29b9c78565be19b14da0183e99d5218b Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Thu, 26 Sep 2024 13:38:42 +0800 Subject: [PATCH 1/7] docs: add FITS into the algo table; --- README.md | 6 +++++- README_zh.md | 6 +++++- docs/index.rst | 2 ++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0556fb10..a80c28a9 100644 --- a/README.md +++ b/README.md @@ -121,6 +121,7 @@ The paper references and links are all listed at the bottom of this file. |:--------------|:---------------------------------------------------------------------------------------------------------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:-------------------| | LLM | Time-Series.AI [^36] | ✅ | ✅ | ✅ | ✅ | ✅ | `Later in 2024` | | Neural Net | TEFN🧑‍🔧[^39] | ✅ | | | | | `2024 - arXiv` | +| Neural Net | FITS🧑‍🔧[^41] | ✅ | | | | | `2024 - ICLR` | | Neural Net | TimeMixer[^37] | ✅ | | | | | `2024 - ICLR` | | Neural Net | iTransformer🧑‍🔧[^24] | ✅ | | | | | `2024 - ICLR` | | Neural Net | ModernTCN[^38] | ✅ | | | | | `2024 - ICLR` | @@ -504,4 +505,7 @@ Time-Series.AI [^39]: Zhan, T., He, Y., Li, Z., & Deng, Y. (2024). [Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting](https://arxiv.org/abs/2405.06419). *arXiv 2024*. -[^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) \ No newline at end of file +[^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) +[^41]: Xu, Z., Zeng, A., & Xu, Q. (2024). +[FITS: Modeling Time Series with 10k parameters](https://openreview.net/forum?id=bWcnvZ3qMb). +*ICLR 2024*. diff --git a/README_zh.md b/README_zh.md index ef47a408..2cce4e12 100644 --- a/README_zh.md +++ b/README_zh.md @@ -106,6 +106,7 @@ PyPOTS当前支持多变量POTS数据的插补, 预测, 分类, 聚类以及异 |:--------------|:---------------------------------------------------------------------------------------------------------------------------------|:------:|:------:|:------:|:------:|:--------:|:-------------------| | LLM | Time-Series.AI [^36] | ✅ | ✅ | ✅ | ✅ | ✅ | `Later in 2024` | | Neural Net | TEFN🧑‍🔧[^39] | ✅ | | | | | `2024 - arXiv` | +| Neural Net | FITS🧑‍🔧[^41] | ✅ | | | | | `2024 - ICLR` | | Neural Net | TimeMixer[^37] | ✅ | | | | | `2024 - ICLR` | | Neural Net | iTransformer🧑‍🔧[^24] | ✅ | | | | | `2024 - ICLR` | | Neural Net | ModernTCN[^38] | ✅ | | | | | `2024 - ICLR` | @@ -477,4 +478,7 @@ Time-Series.AI [^39]: Zhan, T., He, Y., Li, Z., & Deng, Y. (2024). [Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting](https://arxiv.org/abs/2405.06419). *arXiv 2024*. -[^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) \ No newline at end of file +[^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) +[^41]: Xu, Z., Zeng, A., & Xu, Q. (2024). +[FITS: Modeling Time Series with 10k parameters](https://openreview.net/forum?id=bWcnvZ3qMb). +*ICLR 2024*. diff --git a/docs/index.rst b/docs/index.rst index 1d1fe36f..4aaeb762 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -135,6 +135,8 @@ The paper references are all listed at the bottom of this readme file. +================+===========================================================+======+======+======+======+======+=======================+ | Neural Net | TEFN🧑‍🔧 :cite:`zhan2024tefn` | ✅ | | | | | ``2024 - arXiv`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ +| Neural Net | FITS🧑‍🔧 :cite:`xu2024fits` | ✅ | | | | | ``2024 - ICLR`` | ++----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | TimeMixer :cite:`wang2024timemixer` | ✅ | | | | | ``2024 - ICLR`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | iTransformer🧑‍🔧 :cite:`liu2024itransformer` | ✅ | | | | | ``2024 - ICLR`` | From ef3dc0c188a45a2bd22342cb04c260017c2fbf78 Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Thu, 26 Sep 2024 19:39:49 +0800 Subject: [PATCH 2/7] docs: update the reference of TEFN; --- README.md | 2 +- README_zh.md | 2 +- docs/references.bib | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a80c28a9..47ee47fa 100644 --- a/README.md +++ b/README.md @@ -502,7 +502,7 @@ Time-Series.AI [^38]: Luo, D., & Wang X. (2024). [ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis](https://openreview.net/forum?id=vpJMJerXHU). *ICLR 2024*. -[^39]: Zhan, T., He, Y., Li, Z., & Deng, Y. (2024). +[^39]: Zhan, T., He, Y., Deng, Y., Li, Z., Du, W., & Wen, Q. (2024). [Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting](https://arxiv.org/abs/2405.06419). *arXiv 2024*. [^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) diff --git a/README_zh.md b/README_zh.md index 2cce4e12..55978e01 100644 --- a/README_zh.md +++ b/README_zh.md @@ -475,7 +475,7 @@ Time-Series.AI [^38]: Luo, D., & Wang X. (2024). [ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis](https://openreview.net/forum?id=vpJMJerXHU). *ICLR 2024*. -[^39]: Zhan, T., He, Y., Li, Z., & Deng, Y. (2024). +[^39]: Zhan, T., He, Y., Deng, Y., Li, Z., Du, W., & Wen, Q. (2024). [Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting](https://arxiv.org/abs/2405.06419). *arXiv 2024*. [^40]: [Wikipedia: Linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) diff --git a/docs/references.bib b/docs/references.bib index 171d63c5..ce0014ea 100644 --- a/docs/references.bib +++ b/docs/references.bib @@ -766,7 +766,7 @@ @article{bai2018tcn @article{zhan2024tefn, title={Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting}, -author={Zhan, Tianxiang and He, Yuanpeng and Li, Zhen and Deng, Yong}, +author={Zhan, Tianxiang and He, Yuanpeng and Deng, Yong and Li, Zhen and Du, Wenjie and Wen, Qingsong}, journal={arXiv preprint arXiv:2405.06419}, year={2024} } From b6eaa108f21a49385282dec4151b4122de788ca6 Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Fri, 27 Sep 2024 00:53:11 +0800 Subject: [PATCH 3/7] docs: update pytorch intersphinx mapping link; --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 5b8e3822..f21afad7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -62,7 +62,7 @@ intersphinx_mapping = { "python": ("https://docs.python.org/3", None), "sphinx": ("https://www.sphinx-doc.org/en/master", None), - "torch": ("https://pytorch.org/docs/master/", None), + "torch": ("https://pytorch.org/docs/main/", None), "numpy": ("https://numpy.org/doc/stable/", None), "pandas": ("https://pandas.pydata.org/docs/", None), } From c784d802d7871f9f0b1ef49fec75322fdd07b337 Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Fri, 27 Sep 2024 00:53:26 +0800 Subject: [PATCH 4/7] docs: update docs; --- docs/pypots.imputation.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/pypots.imputation.rst b/docs/pypots.imputation.rst index b7a94b03..a7b47b07 100644 --- a/docs/pypots.imputation.rst +++ b/docs/pypots.imputation.rst @@ -19,6 +19,24 @@ pypots.imputation.transformer :show-inheritance: :inherited-members: +pypots.imputation.tefn +------------------------------------ + +.. automodule:: pypots.imputation.tefn + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + +pypots.imputation.fits +------------------------------------ + +.. automodule:: pypots.imputation.fits + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + pypots.imputation.timemixer ------------------------------------ From 6d70c72bc8936f8ea0056259fbfbb52edd1ea295 Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Wed, 16 Oct 2024 18:40:21 +0800 Subject: [PATCH 5/7] docs: update docs for new added models CSAI and SegRNN; --- README.md | 8 ++++++++ README_zh.md | 8 ++++++++ docs/index.rst | 4 ++++ docs/references.bib | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/README.md b/README.md index 47ee47fa..fcfda433 100644 --- a/README.md +++ b/README.md @@ -136,6 +136,8 @@ The paper references and links are all listed at the bottom of this file. | Neural Net | MICN🧑‍🔧[^27] | ✅ | | | | | `2023 - ICLR` | | Neural Net | DLinear🧑‍🔧[^17] | ✅ | | | | | `2023 - AAAI` | | Neural Net | TiDE🧑‍🔧[^28] | ✅ | | | | | `2023 - TMLR` | +| Neural Net | CSAI[^42] | ✅ | | | | | `2023 - arXiv` | +| Neural Net | SegRNN🧑‍🔧[^43] | ✅ | | | | | `2023 - arXiv` | | Neural Net | SCINet🧑‍🔧[^30] | ✅ | | | | | `2022 - NeurIPS` | | Neural Net | Nonstationary Tr.🧑‍🔧[^25] | ✅ | | | | | `2022 - NeurIPS` | | Neural Net | FiLM🧑‍🔧[^22] | ✅ | | | | | `2022 - NeurIPS` | @@ -509,3 +511,9 @@ Time-Series.AI [^41]: Xu, Z., Zeng, A., & Xu, Q. (2024). [FITS: Modeling Time Series with 10k parameters](https://openreview.net/forum?id=bWcnvZ3qMb). *ICLR 2024*. +[^42]: Qian, L., Ibrahim, Z., Ellis, H. L., Zhang, A., Zhang, Y., Wang, T., & Dobson, R. (2023). +[Knowledge Enhanced Conditional Imputation for Healthcare Time-series](https://arxiv.org/abs/2312.16713). +*arXiv 2023*. +[^43]: Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., & Zhang, H. (2023). +[SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting](https://arxiv.org/abs/2308.11200). +*arXiv 2023*. diff --git a/README_zh.md b/README_zh.md index 55978e01..d85af8ba 100644 --- a/README_zh.md +++ b/README_zh.md @@ -121,6 +121,8 @@ PyPOTS当前支持多变量POTS数据的插补, 预测, 分类, 聚类以及异 | Neural Net | MICN🧑‍🔧[^27] | ✅ | | | | | `2023 - ICLR` | | Neural Net | DLinear🧑‍🔧[^17] | ✅ | | | | | `2023 - AAAI` | | Neural Net | TiDE🧑‍🔧[^28] | ✅ | | | | | `2023 - TMLR` | +| Neural Net | CSAI[^42] | ✅ | | | | | `2023 - arXiv` | +| Neural Net | SegRNN🧑‍🔧[^43] | ✅ | | | | | `2023 - arXiv` | | Neural Net | SCINet🧑‍🔧[^30] | ✅ | | | | | `2022 - NeurIPS` | | Neural Net | Nonstationary Tr.🧑‍🔧[^25] | ✅ | | | | | `2022 - NeurIPS` | | Neural Net | FiLM🧑‍🔧[^22] | ✅ | | | | | `2022 - NeurIPS` | @@ -482,3 +484,9 @@ Time-Series.AI [^41]: Xu, Z., Zeng, A., & Xu, Q. (2024). [FITS: Modeling Time Series with 10k parameters](https://openreview.net/forum?id=bWcnvZ3qMb). *ICLR 2024*. +[^42]: Qian, L., Ibrahim, Z., Ellis, H. L., Zhang, A., Zhang, Y., Wang, T., & Dobson, R. (2023). +[Knowledge Enhanced Conditional Imputation for Healthcare Time-series](https://arxiv.org/abs/2312.16713). +*arXiv 2023*. +[^43]: Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., & Zhang, H. (2023). +[SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting](https://arxiv.org/abs/2308.11200). +*arXiv 2023*. diff --git a/docs/index.rst b/docs/index.rst index 4aaeb762..8b9c709c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -165,6 +165,10 @@ The paper references are all listed at the bottom of this readme file. +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | TiDE🧑‍🔧 :cite:`das2023tide` | ✅ | | | | | ``2023 - TMLR`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ +| Neural Net | CSAI :cite:`qian2023knowledge` | ✅ | | | | | ``2023 - arXiv`` | ++----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ +| Neural Net | SegRNN🧑‍🔧 :cite:`lin2023segrnn` | ✅ | | | | | ``2023 - arXiv`` | ++----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | SCINet🧑‍🔧 :cite:`liu2022scinet` | ✅ | | | | | ``2022 - NeurIPS`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | Nonstationary Tr🧑‍🔧 :cite:`liu2022nonstationary` | ✅ | | | | | ``2022 - NeurIPS`` | diff --git a/docs/references.bib b/docs/references.bib index ce0014ea..d74b2ba8 100644 --- a/docs/references.bib +++ b/docs/references.bib @@ -779,3 +779,9 @@ @inproceedings{jin2024timellm url={https://openreview.net/forum?id=Unb5CVPtae} } +@article{qian2023knowledge, +title={Knowledge Enhanced Conditional Imputation for Healthcare Time-series}, +author={Qian, Linglong and Ibrahim, Zina and Ellis, Hugh Logan and Zhang, Ao and Zhang, Yuezhou and Wang, Tao and Dobson, Richard}, +journal={arXiv preprint arXiv:2312.16713}, +year={2023} +} \ No newline at end of file From f870ba11035c9f8fa44a610279302d9119442700 Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Wed, 16 Oct 2024 20:05:01 +0800 Subject: [PATCH 6/7] docs: update docs; --- docs/pypots.imputation.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/pypots.imputation.rst b/docs/pypots.imputation.rst index a7b47b07..82d9df2a 100644 --- a/docs/pypots.imputation.rst +++ b/docs/pypots.imputation.rst @@ -28,6 +28,24 @@ pypots.imputation.tefn :show-inheritance: :inherited-members: +pypots.imputation.casi +------------------------------------ + +.. automodule:: pypots.imputation.casi + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + +pypots.imputation.segrnn +------------------------------------ + +.. automodule:: pypots.imputation.segrnn + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + pypots.imputation.fits ------------------------------------ From e6b1dd3a6065ab5ba5d28314ec3ea681fd8dfe2d Mon Sep 17 00:00:00 2001 From: Wenjie Du Date: Fri, 25 Oct 2024 20:16:17 +0800 Subject: [PATCH 7/7] docs: update docs; --- docs/index.rst | 2 +- docs/pypots.imputation.rst | 4 ++-- docs/references.bib | 2 +- pypots/imputation/csai/model.py | 38 +++++++++++++++++++++++++-------- 4 files changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 8b9c709c..fd3b9219 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -165,7 +165,7 @@ The paper references are all listed at the bottom of this readme file. +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | TiDE🧑‍🔧 :cite:`das2023tide` | ✅ | | | | | ``2023 - TMLR`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ -| Neural Net | CSAI :cite:`qian2023knowledge` | ✅ | | | | | ``2023 - arXiv`` | +| Neural Net | CSAI :cite:`qian2023csai` | ✅ | | | | | ``2023 - arXiv`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ | Neural Net | SegRNN🧑‍🔧 :cite:`lin2023segrnn` | ✅ | | | | | ``2023 - arXiv`` | +----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+ diff --git a/docs/pypots.imputation.rst b/docs/pypots.imputation.rst index 82d9df2a..73e55646 100644 --- a/docs/pypots.imputation.rst +++ b/docs/pypots.imputation.rst @@ -28,10 +28,10 @@ pypots.imputation.tefn :show-inheritance: :inherited-members: -pypots.imputation.casi +pypots.imputation.csai ------------------------------------ -.. automodule:: pypots.imputation.casi +.. automodule:: pypots.imputation.csai :members: :undoc-members: :show-inheritance: diff --git a/docs/references.bib b/docs/references.bib index d74b2ba8..10a9632c 100644 --- a/docs/references.bib +++ b/docs/references.bib @@ -779,7 +779,7 @@ @inproceedings{jin2024timellm url={https://openreview.net/forum?id=Unb5CVPtae} } -@article{qian2023knowledge, +@article{qian2023csai, title={Knowledge Enhanced Conditional Imputation for Healthcare Time-series}, author={Qian, Linglong and Ibrahim, Zina and Ellis, Hugh Logan and Zhang, Ao and Zhang, Yuezhou and Wang, Tao and Dobson, Richard}, journal={arXiv preprint arXiv:2312.16713}, diff --git a/pypots/imputation/csai/model.py b/pypots/imputation/csai/model.py index b61286c5..010561a2 100644 --- a/pypots/imputation/csai/model.py +++ b/pypots/imputation/csai/model.py @@ -21,7 +21,8 @@ class CSAI(BaseNNImputer): - """ + """The PyTorch implementation of the CSAI model :cite:`qian2023csai`. + Parameters ---------- n_steps : @@ -58,29 +59,48 @@ class CSAI(BaseNNImputer): The number of epochs for training the model. patience : - The patience for the early-stopping mechanism. Given a positive integer, training will stop when no improvement is observed after the specified number of epochs. If set to None, early-stopping is disabled. + The patience for the early-stopping mechanism. Given a positive integer, the training process will be + stopped when the model does not perform better after that number of epochs. + Leaving it default as None will disable the early-stopping. optimizer : - The optimizer used for model training. Defaults to the Adam optimizer if not specified. + The optimizer for model training. + If not given, will use a default Adam optimizer. num_workers : - The number of subprocesses used for data loading. Setting this to `0` means that data loading is performed in the main process without using subprocesses. + The number of subprocesses to use for data loading. + `0` means data loading will be in the main process, i.e. there won't be subprocesses. device : - The device for the model to run on, which can be a string, a :class:`torch.device` object, or a list of devices. If not provided, the model will attempt to use available CUDA devices first, then default to CPUs. + The device for the model to run on. It can be a string, a :class:`torch.device` object, or a list of them. + If not given, will try to use CUDA devices first (will use the default CUDA device if there are multiple), + then CPUs, considering CUDA and CPU are so far the main devices for people to train ML models. + If given a list of devices, e.g. ['cuda:0', 'cuda:1'], or [torch.device('cuda:0'), torch.device('cuda:1')] , the + model will be parallely trained on the multiple devices (so far only support parallel training on CUDA devices). + Other devices like Google TPU and Apple Silicon accelerator MPS may be added in the future. saving_path : - The path for saving model checkpoints and tensorboard files during training. If not provided, models will not be saved automatically. + The path for automatically saving model checkpoints and tensorboard files (i.e. loss values recorded during + training into a tensorboard file). Will not save if not given. model_saving_strategy : - The strategy for saving model checkpoints. Can be one of [None, "best", "better", "all"]. "best" saves the best model after training, "better" saves any model that improves during training, and "all" saves models after each epoch. If set to None, no models will be saved. + The strategy to save model checkpoints. It has to be one of [None, "best", "better", "all"]. + No model will be saved when it is set as None. + The "best" strategy will only automatically save the best model after the training finished. + The "better" strategy will automatically save the model during training whenever the model performs + better than in previous epochs. + The "all" strategy will save every model after each epoch training. verbose : - Whether to print training logs during the training process. + Whether to print out the training logs during the training process. Notes ----- - CSAI (Consistent Sequential Imputation) is a bidirectional model designed for time-series imputation. It employs a forward and backward GRU network to handle missing data, using consistency and reconstruction losses to improve accuracy. The model supports various training configurations, such as interval computations, early-stopping, and multiple devices for training. Results can be saved based on the specified saving strategy, and tensorboard files are generated for tracking the model's performance over time. + CSAI (Consistent Sequential Imputation) is a bidirectional model designed for time-series imputation. + It employs a forward and backward GRU network to handle missing data, using consistency and reconstruction losses + to improve accuracy. The model supports various training configurations, such as interval computations, + early-stopping, and multiple devices for training. Results can be saved based on the specified saving strategy, + and tensorboard files are generated for tracking the model's performance over time. """