diff --git a/pic/framework.png b/pic/framework.png new file mode 100644 index 0000000..ee90269 Binary files /dev/null and b/pic/framework.png differ diff --git a/pic/workflow.png b/pic/workflow.png new file mode 100644 index 0000000..31380a2 Binary files /dev/null and b/pic/workflow.png differ diff --git a/reademe.md b/reademe.md new file mode 100644 index 0000000..2907326 --- /dev/null +++ b/reademe.md @@ -0,0 +1,82 @@ +# UP2ME: Univariate Pre-training to Multivariate Fine-tuning as a General-purpose Framework for Multivariate Time Series Analysis (ICML 2024) + +This is the origin Pytorch implementation of “UP2ME: Univariate Pre-training to Multivariate Fine-tuning as a General-purpose Framework for Multivariate Time Series Analysis （ICML L2024)” + +## Workflow +UP2ME is a general-purpose framework for Multivariate Time Series Analysis. It conducts taskagnostic pre-training when downstream tasks are unspecified. Once the task and setting (e.g. forecasting length) are determined, it gives sensible solutions with frozen pre-trained parameters. Further accuracy is achieved through multivariate fine-tuning. + +

+ +

+ +## Keypoints + +

+ + +Overview of UP2ME framework. Left: Univariate Pre-training. Univariate instances are generated using variable window length and channel decoupling. Generated instances are fed into the encoder and decoder for Masked AutoEnncoder (MAE) pre-training. Formulating downstream tasks as specific mask-reconstruction problems, UP2ME can give sensible solutions without parameter modification (right part without TC layers). Right: Multivariate Fine-tuning (forecasting in this example). The pre-trained frozen encoder encodes a multivariate series into latent tokens. +The tokens are used to construct a dependency graph among channels. Learnable Temporal-Channel (TC) layers which take constructed graph as input, are inserted before the frozen decoder for fine-tuning. +

+ +### Univariate Pretraining + +1. **Variable Window Length**: To meet the uncertain requirements for window length, for each pre-training step, we randomly sample a window length $L$ then generate a batch of instances with this length. + +2. **Channel Decoupling**: To generate an instance of length $L$, instead of extracting a multivariate sub-series, we independently sample a time span and a channel index to generate a univariate sub-series. + +3. **Immediate Reaction Mode**: After pre-training, UP2ME can perform +immediate forecasting, anomaly detection and imputation with frozen parameters by formulating them into specific mask-reconstruction problems: + + 3.1. **Forecasting**: Past series are viewed as unmasked patches and future series are viewed as masked patches to reconstruct. + + 3.2. **Imputation**: Fully-observed patches are viewed as unmasked patches and patches containing at least one missing point are viewed as masked. + + 3.3 **Anomaly Detection**: Iteratively mask each patch and use other unmasked patches to reconstruct it. Difference between reconstructed series and original series is used as the anomaly score. + +### Multivariate Fine-tuning + +1. **Sparse Dependency Graph Construction**: A sparse dependency graph is constructed using representations output by the pre-trained encoder to guide cross-channel dependency capturing. + +2. **Temporal-Channel (TC) layer**: We freeze parameters of the pre-trained encoder and decoder while inserting learnable Temporal-Channel (TC) layers between them to capture cross-channel dependency and adjust temporal dependency. With few inductive biases, our TC layer contains a standard Transformer layer and a standard +Graph Transformer layer. The Graph Transformer layer takes the constructed dependency graph as input. + +## Reproducibility +1. Install requirements by: + + ``` + pip install -r requirements.txt + ``` + + +2. Download the datasets from [UP2ME-datasets](https://drive.google.com/file/d/1oLYcQa7NJcMDSP_rYSkP5hQHzXL2rpZM/view?usp=drive_link) and unzip it into the folder `datasets` in the root folder. The struture should be like: + + - datasets + - ETT + - ETTm1.csv + - ...... same for csv format datasets: weather, ECL(Electricity) and traffic + - SMD + - SMD_train.npy + - SMD_test.npy + - SMD_test_label.npy + - ...... same for npy format datasets: PSM, SWaT and NIPS_Water(GECCO) + + + +3. We have already put the pre-trained model for each dataset in `./pretrain-library`. To get forecasting results on ETTm1, run: + ``` + bash scripts/forecast_scripts/ETTm1.sh + ``` + the immediate reaction(UP2ME(IR)) and fine-tuning(UP2ME(FT)) modes will be tested on 4 different forecasting lengths (96, 192, 336, 720) and results will be saved in a new folder `./forecast_result`. + +4. To reproduce results for all 3 tasks on all 8 datasets, run other scripts in `./scripts`. + +## Acknowledgement +We appreciate the following works for their valuable code and data: + +https://github.com/thuml/Time-Series-Library + +https://github.com/DAMO-DI-ML/KDD2023-DCdetector + +https://github.com/zezhishao/STEP + +https://github.com/facebookresearch/mae diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..810e484 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,7 @@ +einops==0.8.0 +loguru==0.7.2 +numpy==1.24.4 +pandas==2.0.3 +scikit_learn==1.3.0 +torch==2.0.1 +torch_geometric==2.3.1