diff --git a/.gitignore b/.gitignore index 552c717..bf34f9e 100755 --- a/.gitignore +++ b/.gitignore @@ -152,3 +152,4 @@ checkpoints/* *.png *.nc *.zarr +*.slurm diff --git a/README.md b/README.md index c3fa260..fccb5f4 100755 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ We recommend using Anaconda for managing dependencies of this library. The follo conda create -n emp -c conda-forge python conda activate emp pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 -pip install numpy matplotlib pillow xarray zarr netcdf4 scipy imageio networkx scikit-image s3fs pyproj pyarrow seaborn cartopy tensorboard dask pytorch-lightning torchmetrics statsmodels earthnet earthnet-minicuber segmentation-models-pytorch albumentations ipykernel +pip install numpy matplotlib pillow xarray zarr netcdf4 scipy imageio networkx scikit-image s3fs pyproj pyarrow seaborn cartopy tensorboard dask pytorch-lightning torchmetrics statsmodels earthnet earthnet-minicuber segmentation-models-pytorch albumentations ipykernel geopandas ``` ## Installation diff --git a/configs/de23/convlstm_ae/all_variables.yaml b/configs/de23/convlstm_ae/all_variables.yaml new file mode 100755 index 0000000..e7841df --- /dev/null +++ b/configs/de23/convlstm_ae/all_variables.yaml @@ -0,0 +1,84 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-3.yaml b/configs/de23/convlstm_ae/lr_1e-3.yaml new file mode 100755 index 0000000..5cf9079 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-3.yaml @@ -0,0 +1,84 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + #profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_continent_1.yaml b/configs/de23/convlstm_ae/lr_1e-4_continent_1.yaml new file mode 100644 index 0000000..2cabe02 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_continent_1.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_continent_2.yaml b/configs/de23/convlstm_ae/lr_1e-4_continent_2.yaml new file mode 100644 index 0000000..31f343a --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_continent_2.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 8 + val_fold: 7 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_continent_3.yaml b/configs/de23/convlstm_ae/lr_1e-4_continent_3.yaml new file mode 100644 index 0000000..4a5fdab --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_continent_3.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 6 + val_fold: 5 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_continent_4.yaml b/configs/de23/convlstm_ae/lr_1e-4_continent_4.yaml new file mode 100644 index 0000000..c7e56f3 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_continent_4.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 4 + val_fold: 3 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_continent_5.yaml b/configs/de23/convlstm_ae/lr_1e-4_continent_5.yaml new file mode 100644 index 0000000..b62d49e --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_continent_5.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 2 + val_fold: 1 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_iid_1.yaml b/configs/de23/convlstm_ae/lr_1e-4_iid_1.yaml new file mode 100644 index 0000000..04aa89f --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_iid_1.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_iid_2.yaml b/configs/de23/convlstm_ae/lr_1e-4_iid_2.yaml new file mode 100644 index 0000000..c5037a9 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_iid_2.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 8 + val_fold: 7 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_iid_3.yaml b/configs/de23/convlstm_ae/lr_1e-4_iid_3.yaml new file mode 100644 index 0000000..60f5d93 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_iid_3.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 6 + val_fold: 5 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_iid_4.yaml b/configs/de23/convlstm_ae/lr_1e-4_iid_4.yaml new file mode 100644 index 0000000..6223ffe --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_iid_4.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 4 + val_fold: 3 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_iid_5.yaml b/configs/de23/convlstm_ae/lr_1e-4_iid_5.yaml new file mode 100644 index 0000000..4fcf42f --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_iid_5.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 10 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 2 + val_fold: 1 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [1, 7] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_mask.yaml b/configs/de23/convlstm_ae/lr_1e-4_mask.yaml new file mode 100644 index 0000000..816e05e --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_mask.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + is_lc_mask: True + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] # never used [3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_mlst50.yaml b/configs/de23/convlstm_ae/lr_1e-4_mlst50.yaml new file mode 100755 index 0000000..72951fe --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_mlst50.yaml @@ -0,0 +1,83 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-4_nolc.yaml b/configs/de23/convlstm_ae/lr_1e-4_nolc.yaml new file mode 100644 index 0000000..35244e7 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-4_nolc.yaml @@ -0,0 +1,86 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + # lc_mask: False + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + is_lc_mask: False + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.0001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] # never used [3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-5.yaml b/configs/de23/convlstm_ae/lr_1e-5.yaml new file mode 100755 index 0000000..ca493e0 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-5.yaml @@ -0,0 +1,84 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.00001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50] # [3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-5_continent.yaml b/configs/de23/convlstm_ae/lr_1e-5_continent.yaml new file mode 100644 index 0000000..a757e63 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-5_continent.yaml @@ -0,0 +1,85 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + continent_split: True + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.00001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-5_interpolated.yaml b/configs/de23/convlstm_ae/lr_1e-5_interpolated.yaml new file mode 100644 index 0000000..30ad104 --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-5_interpolated.yaml @@ -0,0 +1,82 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 73 + target_length: 17 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.00001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50,90] #[3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/de23/convlstm_ae/lr_1e-6.yaml b/configs/de23/convlstm_ae/lr_1e-6.yaml new file mode 100755 index 0000000..705507c --- /dev/null +++ b/configs/de23/convlstm_ae/lr_1e-6.yaml @@ -0,0 +1,84 @@ +Architecture: "convlstm_ae" + +Seed: 42 + +Setting: "de23" + +Logger: + save_dir: "experiments/" + + +Checkpointer: + save_top_k: 1 + save_last: True + every_n_epochs: 1 + +Trainer: + gpus: 1 + strategy: 'ddp_find_unused_parameters_false' + #deterministic: True + log_every_n_steps: 32 + profiler: 'simple' + accumulate_grad_batches: 8 + #fast_dev_run: True + #log_gpu_memory: 'all' + #weights_summary: 'full' + max_epochs: 30 + #limit_train_batches: 32 + #limit_val_batches: 32 + gradient_clip_val: 1 + # check_val_every_n_epoch: 1 # default setting + # Whether to use torch.inference_mode() or torch.no_grad() mode during evaluation + # inference_mode: True # if False use 'torch.no_grad' instead. + val_check_interval: 0.25 #0.005 # + +Data: + base_dir: "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes" # "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" # + test_fold: 10 + val_fold: 9 + fold_path: "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv" + test_track: "iid" + target: "ndvi" + train_batch_size: 4 + val_batch_size: 4 + test_batch_size: 4 + num_workers: 4 + + +Task: + loss: + name: "MaskedPixelwiseLoss" + # lc_min: 40 + # lc_max: 90 + ndvi_pred_idx: 0 # index of the NDVI band + ndvi_targ_idx: 0 # index of the NDVI band + pred_mask_value: -1 + scale_by_std: False + context_length: 72 + target_length: 18 + metric: "RMSE" + n_stochastic_preds: 2 + optimization: + optimizer: + - + name: AdamW + args: + betas: [0.9, 0.999] + lr_per_sample: 0.000001 + lr_shedule: + - + name: MultiStepLR + args: + milestones: [50] # [3, 6] #[2, 20, 50, 90] + gamma: 0.1 + n_log_batches: 2 + compute_metric_on_test: True + +Model: + hidden_dim: [64, 64, 64, 64] + kernel_size: 3 + bias: True + skip_connections: True + num_inputs: 33 # 1 (target) + 7 (s2) + 24 (meteo) + 1 (topo) + num_outputs: 1 + teacher_forcing: True \ No newline at end of file diff --git a/configs/en21x/convlstm/tune_basic/lr_0.1.yaml b/configs/en21x/convlstm/tune_basic/lr_0.1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/convlstm/tune_basic/lr_1.yaml b/configs/en21x/convlstm/tune_basic/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/convlstm/tune_basic/lr_10.yaml b/configs/en21x/convlstm/tune_basic/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_resnet/tune_basic/lr_1_1x1.yaml b/configs/en21x/nf_resnet/tune_basic/lr_1_1x1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_resnet/tune_basic/lr_1_3x3.yaml b/configs/en21x/nf_resnet/tune_basic/lr_1_3x3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_basic/lr_10.yaml b/configs/en21x/nf_unet/tune_basic/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_weather/lr_10_film_latent.yaml b/configs/en21x/nf_unet/tune_weather/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_weather/lr_10_film_latent_shuffle.yaml b/configs/en21x/nf_unet/tune_weather/lr_10_film_latent_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_weather/lr_10_film_latent_shuffle2.yaml b/configs/en21x/nf_unet/tune_weather/lr_10_film_latent_shuffle2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_weather/lr_10_noweather.yaml b/configs/en21x/nf_unet/tune_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/nf_unet/tune_weather/lr_10_xattn_latent.yaml b/configs/en21x/nf_unet/tune_weather/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_action.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_action.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_cat_all.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_cat_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_cat_early.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_cat_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_cat_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_all.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_early.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_early_2.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_early_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_early_3.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_early_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_film_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_film_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_noweather.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_xattn_all.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_xattn_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_xattn_early.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_xattn_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_xattn_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_10_xattn_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_10_xattn_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_20_film_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_20_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_20_film_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_20_film_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_20_xattn_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_20_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_20_xattn_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_20_xattn_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_5_film_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_5_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_5_film_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_5_film_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_5_xattn_latent.yaml b/configs/en21x/predrnn/ablate_weather/lr_5_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/ablate_weather/lr_5_xattn_latenth.yaml b/configs/en21x/predrnn/ablate_weather/lr_5_xattn_latenth.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_1.25_hr_cat_early.yaml b/configs/en21x/predrnn/extra_exp/lr_1.25_hr_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_10_film_early_shuffle.yaml b/configs/en21x/predrnn/extra_exp/lr_10_film_early_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_10_hr_cat_early.yaml b/configs/en21x/predrnn/extra_exp/lr_10_hr_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_2.5_film_early.yaml b/configs/en21x/predrnn/extra_exp/lr_2.5_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_2.5_film_early_shuffle.yaml b/configs/en21x/predrnn/extra_exp/lr_2.5_film_early_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/extra_exp/lr_5_hr_cat_early.yaml b/configs/en21x/predrnn/extra_exp/lr_5_hr_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_early.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_earlyS.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_earlyS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_earlyS_2.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_earlyS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_earlyS_3.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_earlyS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_early_2.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_early_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_early_3.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_early_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_latent.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_latent_2.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_latent_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_film_latent_3.yaml b/configs/en21x/predrnn/lastrun/lr_10_film_latent_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweather.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweatherS.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweatherS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweatherS_2.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweatherS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweatherS_3.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweatherS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweather_2.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweather_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun/lr_10_noweather_3.yaml b/configs/en21x/predrnn/lastrun/lr_10_noweather_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS.yaml b/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS_2.yaml b/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS_3.yaml b/configs/en21x/predrnn/lastrun2/lr_10_film_earlyS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_basic/lr_0.1.yaml b/configs/en21x/predrnn/tune_basic/lr_0.1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_basic/lr_1.yaml b/configs/en21x/predrnn/tune_basic/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_basic/lr_10.yaml b/configs/en21x/predrnn/tune_basic/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_patch/lr_1.yaml b/configs/en21x/predrnn/tune_patch/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_patch/lr_10.yaml b/configs/en21x/predrnn/tune_patch/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_patch/lr_10_none.yaml b/configs/en21x/predrnn/tune_patch/lr_10_none.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_patch/lr_1_none.yaml b/configs/en21x/predrnn/tune_patch/lr_1_none.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_scalebystd/lr_1.yaml b/configs/en21x/predrnn/tune_scalebystd/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_scalebystd/lr_1_dlmask.yaml b/configs/en21x/predrnn/tune_scalebystd/lr_1_dlmask.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_0.1_l2.yaml b/configs/en21x/predrnn/tune_v2/lr_0.1_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_1.yaml b/configs/en21x/predrnn/tune_v2/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_10_l2.yaml b/configs/en21x/predrnn/tune_v2/lr_10_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_1_l2.yaml b/configs/en21x/predrnn/tune_v2/lr_1_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_1_l2_relubn.yaml b/configs/en21x/predrnn/tune_v2/lr_1_l2_relubn.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_1_l2_xaux.yaml b/configs/en21x/predrnn/tune_v2/lr_1_l2_xaux.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_v2/lr_1_l2_xaux_xshed.yaml b/configs/en21x/predrnn/tune_v2/lr_1_l2_xaux_xshed.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather/lr_10_noweather.yaml b/configs/en21x/predrnn/tune_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_action.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_action.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_cat_all.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_cat_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_cat_early.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_cat_latent.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_cat_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_film_all.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_film_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_film_early.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_film_latent.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_film_latent_h.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_film_latent_h.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_noweather.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_xattn_all.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_xattn_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_xattn_early.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_xattn_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_xattn_early2.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_xattn_early2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_xattn_latent.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_weather2/lr_10_xattn_latent_h.yaml b/configs/en21x/predrnn/tune_weather2/lr_10_xattn_latent_h.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_wstatic/lr_0.1.yaml b/configs/en21x/predrnn/tune_wstatic/lr_0.1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_wstatic/lr_1.yaml b/configs/en21x/predrnn/tune_wstatic/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/predrnn/tune_wstatic/lr_10.yaml b/configs/en21x/predrnn/tune_wstatic/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_cat_all.yaml b/configs/en21x/simvp/ablate_weather/lr_10_cat_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_cat_early.yaml b/configs/en21x/simvp/ablate_weather/lr_10_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_cat_latent.yaml b/configs/en21x/simvp/ablate_weather/lr_10_cat_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_cat_latentproc.yaml b/configs/en21x/simvp/ablate_weather/lr_10_cat_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_all.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_early.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_latent.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc_2.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc_3.yaml b/configs/en21x/simvp/ablate_weather/lr_10_film_latentproc_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_noweather.yaml b/configs/en21x/simvp/ablate_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_xattn_all.yaml b/configs/en21x/simvp/ablate_weather/lr_10_xattn_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_xattn_early.yaml b/configs/en21x/simvp/ablate_weather/lr_10_xattn_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_xattn_latent.yaml b/configs/en21x/simvp/ablate_weather/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_10_xattn_latentproc.yaml b/configs/en21x/simvp/ablate_weather/lr_10_xattn_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_5_film_latent.yaml b/configs/en21x/simvp/ablate_weather/lr_5_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_5_film_latentproc.yaml b/configs/en21x/simvp/ablate_weather/lr_5_film_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_5_xattn_latent.yaml b/configs/en21x/simvp/ablate_weather/lr_5_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/ablate_weather/lr_5_xattn_latentproc.yaml b/configs/en21x/simvp/ablate_weather/lr_5_xattn_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/extra_exp/lr_1.25_film_latentproc.yaml b/configs/en21x/simvp/extra_exp/lr_1.25_film_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/extra_exp/lr_1.25_film_latentproc_shuffle.yaml b/configs/en21x/simvp/extra_exp/lr_1.25_film_latentproc_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/extra_exp/lr_10_film_latentproc_shuffle.yaml b/configs/en21x/simvp/extra_exp/lr_10_film_latentproc_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentproc.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentprocS.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentprocS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentprocS_2.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentprocS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentprocS_3.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentprocS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentproc_2.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentproc_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_latentproc_3.yaml b/configs/en21x/simvp/lastrun/lr_10_film_latentproc_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_noweatherS.yaml b/configs/en21x/simvp/lastrun/lr_10_film_noweatherS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_noweatherS_2.yaml b/configs/en21x/simvp/lastrun/lr_10_film_noweatherS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun/lr_10_film_noweatherS_3.yaml b/configs/en21x/simvp/lastrun/lr_10_film_noweatherS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS.yaml b/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS_2.yaml b/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS_3.yaml b/configs/en21x/simvp/lastrun2/lr_10_film_latentprocS_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_film_latentproc.yaml b/configs/en21x/simvp/oldval/lr_10_film_latentproc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_film_latentproc_2.yaml b/configs/en21x/simvp/oldval/lr_10_film_latentproc_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_film_latentproc_3.yaml b/configs/en21x/simvp/oldval/lr_10_film_latentproc_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_film_latentproc_saveall.yaml b/configs/en21x/simvp/oldval/lr_10_film_latentproc_saveall.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_noweather.yaml b/configs/en21x/simvp/oldval/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_noweather_2.yaml b/configs/en21x/simvp/oldval/lr_10_noweather_2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/oldval/lr_10_noweather_3.yaml b/configs/en21x/simvp/oldval/lr_10_noweather_3.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_basic/lr_0.1.yaml b/configs/en21x/simvp/tune_basic/lr_0.1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_basic/lr_1.yaml b/configs/en21x/simvp/tune_basic/lr_1.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_basic/lr_10.yaml b/configs/en21x/simvp/tune_basic/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_basic/lr_100.yaml b/configs/en21x/simvp/tune_basic/lr_100.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_encoder/lr_10_patch_shuffletrain_longer.yaml b/configs/en21x/simvp/tune_encoder/lr_10_patch_shuffletrain_longer.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_L_film.yaml b/configs/en21x/simvp/tune_proc/gsta_L_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_L_film_l2.yaml b/configs/en21x/simvp/tune_proc/gsta_L_film_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film_l2.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film_w16_bs16.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film_w16_bs16.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film_w32.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film_w32.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film_w4.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film_w4.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/gsta_S_film_w8.yaml b/configs/en21x/simvp/tune_proc/gsta_S_film_w8.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/incep_L_film.yaml b/configs/en21x/simvp/tune_proc/incep_L_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/incep_L_film_l2.yaml b/configs/en21x/simvp/tune_proc/incep_L_film_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/incep_S_film.yaml b/configs/en21x/simvp/tune_proc/incep_S_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_proc/incep_S_film_l2.yaml b/configs/en21x/simvp/tune_proc/incep_S_film_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_scalebystd/lr_10.yaml b/configs/en21x/simvp/tune_scalebystd/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_scalebystd/lr_10_dlmask.yaml b/configs/en21x/simvp/tune_scalebystd/lr_10_dlmask.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_100_cat.yaml b/configs/en21x/simvp/tune_v2/lr_100_cat.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_100_film.yaml b/configs/en21x/simvp/tune_v2/lr_100_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_100_none.yaml b/configs/en21x/simvp/tune_v2/lr_100_none.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_cat.yaml b/configs/en21x/simvp/tune_v2/lr_10_cat.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_film.yaml b/configs/en21x/simvp/tune_v2/lr_10_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_film_l2.yaml b/configs/en21x/simvp/tune_v2/lr_10_film_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_film_lc.yaml b/configs/en21x/simvp/tune_v2/lr_10_film_lc.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_none.yaml b/configs/en21x/simvp/tune_v2/lr_10_none.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_10_none_l2.yaml b/configs/en21x/simvp/tune_v2/lr_10_none_l2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_1_cat.yaml b/configs/en21x/simvp/tune_v2/lr_1_cat.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_1_film.yaml b/configs/en21x/simvp/tune_v2/lr_1_film.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_v2/lr_1_none.yaml b/configs/en21x/simvp/tune_v2/lr_1_none.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather/lr_10_cat.yaml b/configs/en21x/simvp/tune_weather/lr_10_cat.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather/lr_10_noweather.yaml b/configs/en21x/simvp/tune_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather/lr_10_xattn.yaml b/configs/en21x/simvp/tune_weather/lr_10_xattn.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_cat_all.yaml b/configs/en21x/simvp/tune_weather2/lr_10_cat_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_cat_early.yaml b/configs/en21x/simvp/tune_weather2/lr_10_cat_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_cat_latent.yaml b/configs/en21x/simvp/tune_weather2/lr_10_cat_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_film_all.yaml b/configs/en21x/simvp/tune_weather2/lr_10_film_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_film_early.yaml b/configs/en21x/simvp/tune_weather2/lr_10_film_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_film_latent.yaml b/configs/en21x/simvp/tune_weather2/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_film_latent_val2.yaml b/configs/en21x/simvp/tune_weather2/lr_10_film_latent_val2.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_film_latent_val2_nofg.yaml b/configs/en21x/simvp/tune_weather2/lr_10_film_latent_val2_nofg.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_noweather_long.yaml b/configs/en21x/simvp/tune_weather2/lr_10_noweather_long.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_xattn_all.yaml b/configs/en21x/simvp/tune_weather2/lr_10_xattn_all.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_xattn_early.yaml b/configs/en21x/simvp/tune_weather2/lr_10_xattn_early.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_xattn_latent.yaml b/configs/en21x/simvp/tune_weather2/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/simvp/tune_weather2/lr_10_xattn_latent_mlp.yaml b/configs/en21x/simvp/tune_weather2/lr_10_xattn_latent_mlp.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/st_unet/tune_weather/lr_10_film_latent.yaml b/configs/en21x/st_unet/tune_weather/lr_10_film_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/st_unet/tune_weather/lr_10_film_latent_shuffle.yaml b/configs/en21x/st_unet/tune_weather/lr_10_film_latent_shuffle.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/st_unet/tune_weather/lr_10_noweather.yaml b/configs/en21x/st_unet/tune_weather/lr_10_noweather.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/st_unet/tune_weather/lr_10_xattn_latent.yaml b/configs/en21x/st_unet/tune_weather/lr_10_xattn_latent.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/unet_lstm/tune_weather/lr_10.yaml b/configs/en21x/unet_lstm/tune_weather/lr_10.yaml old mode 100644 new mode 100755 diff --git a/configs/en21x/unet_lstm/tune_weather/lr_10_gru.yaml b/configs/en21x/unet_lstm/tune_weather/lr_10_gru.yaml old mode 100644 new mode 100755 diff --git a/configs/en23/convlstm_ae/all_variables.yaml b/configs/en23/convlstm_ae/all_variables.yaml index eb42596..422b8bc 100755 --- a/configs/en23/convlstm_ae/all_variables.yaml +++ b/configs/en23/convlstm_ae/all_variables.yaml @@ -33,9 +33,9 @@ Trainer: val_check_interval: 0.25 #0.005 # Data: - base_dir: "/scratch/crobin/earthnet2023/" + base_dir: "/scratch/crobin/earthnet2023_preprocessing/" test_track: "iid" - target: "anomalie_ndvi" + target: "ndvi" train_batch_size: 4 val_batch_size: 4 test_batch_size: 4 diff --git a/earthnet_models_pytorch/data/__init__.py b/earthnet_models_pytorch/data/__init__.py index 94e507d..8d8fd8e 100755 --- a/earthnet_models_pytorch/data/__init__.py +++ b/earthnet_models_pytorch/data/__init__.py @@ -1,3 +1,4 @@ +from earthnet_models_pytorch.data.de23_data import DeepExtremes2023DataModule from earthnet_models_pytorch.data.en21_data import EarthNet2021DataModule from earthnet_models_pytorch.data.en21x_data import EarthNet2021XDataModule from earthnet_models_pytorch.data.en21x_data_old import ( @@ -17,6 +18,7 @@ "en21x-pxold", "en22", "en23", + "de23", ] DATASETS = { @@ -28,6 +30,7 @@ "en21x-pxold": EarthNet2021XpxOldDataModule, "en22": EarthNet2022DataModule, "en23": EarthNet2023DataModule, + "de23": DeepExtremes2023DataModule, } METRIC_CHECKPOINT_INFO = { @@ -71,4 +74,9 @@ "filename": "Epoch-{epoch:02d}-RMSE (Vegetation)-{RMSE_Veg:.4f}", "mode": "min", }, + "de23": { + "monitor": "RMSE_Veg", + "filename": "Epoch-{epoch:02d}-RMSE (Vegetation)-{RMSE_Veg:.4f}", + "mode": "min", + }, } diff --git a/earthnet_models_pytorch/data/de23_data.py b/earthnet_models_pytorch/data/de23_data.py new file mode 100755 index 0000000..1dfc680 --- /dev/null +++ b/earthnet_models_pytorch/data/de23_data.py @@ -0,0 +1,603 @@ +from typing import Union, Optional + +import argparse +import copy +import multiprocessing +import pandas as pd +import geopandas as gpd +from shapely.geometry import Point +from pathlib import Path + +import numpy as np +import pytorch_lightning as pl +import torch +import xarray as xr +import datetime + +from torch.utils.data import Dataset, DataLoader, random_split + +from earthnet_models_pytorch.utils import str2bool + +import json + + +variables = { + ### variables with dimensions x, y, Sampled time (irregular) + # Sentinel 2 bands. Spatio-temporal data. Every 5 days. + "s2_bands": ["B02", "B03", "B04", "B05", "B06", "B07", "B8A"], + # "s2_avail": ["s2_avail"], + "s2_scene_classification": ["SCL"], + # S2 EarthNet cloudmask + "cloud_mask": ["cloudmask_en"], + ### variables with dimension time + # Era5 reanalysis dataset. Era5 pev has a low precision + "era5": [ + "t2m_mean", + "pev_mean", + "slhf_mean", + "ssr_mean", + "sp_mean", + "sshf_mean", + "e_mean", + "tp_mean", + "t2m_min", + "pev_min", + "slhf_min", + "ssr_min", + "sp_min", + "sshf_min", + "e_min", + "tp_min", + "t2m_max", + "pev_max", + "slhf_max", + "ssr_max", + "sp_max", + "sshf_max", + "e_max", + "tp_max", + ], + ### variables with dimension event_time + "event": [ + "events", + "event_labels", + ], + ### static variables + ### variables with dimensions x_300, y_300, + # Land cover classes. Categorical variable. + # use Scene classification instead + "landcover": ["lccs_class"], + ### variables with dimensions x, y + # Elevation model. Defined on 0 - 2000. + "elevation": ["cop_dem"], +} + +# with open("./scripts/preprocessing/statistics_de23.json", "r") as f: +# statistic = json.load(f) +statistic = { + "t2m_mean": { + "mean": 285.14280428589245, + "min": 220.7626190185547, + "max": 318.43597412109375, + }, + "pev_mean": { + "mean": -0.003933192138161199, + "min": -0.049169037491083145, + "max": 0.0005616411217488348, + }, + "slhf_mean": {"mean": -2162931.7410918274, "min": -15086400.0, "max": 2074217.375}, + "ssr_mean": {"mean": 6422307.595667305, "min": -0.0625, "max": 22510174.0}, + "sp_mean": {"mean": 91747.36897021797, "min": 49799.546875, "max": 105716.3984375}, + "sshf_mean": {"mean": -1334688.0814032578, "min": -12454722.0, "max": 9606654.0}, + "e_mean": { + "mean": -0.0008624358549881119, + "min": -0.006032629404217005, + "max": 0.0007265469757840037, + }, + "tp_mean": {"mean": 0.0015462739210485358, "min": 0.0, "max": 0.09146501123905182}, + "t2m_min": { + "mean": 278.7418223321622, + "min": 214.3935089111328, + "max": 310.7576904296875, + }, + "pev_min": { + "mean": -0.010466829879496528, + "min": -0.1887352466583252, + "max": 1.1321157217025757e-05, + }, + "slhf_min": {"mean": -5363088.528838599, "min": -45605476.0, "max": 163808.0}, + "ssr_min": {"mean": 147228.9512738576, "min": -1.0, "max": 3258205.0}, + "sp_min": {"mean": 91157.6636795228, "min": 49396.6875, "max": 105117.6875}, + "sshf_min": {"mean": -4187134.392026041, "min": -26917700.0, "max": 742063.0}, + "e_min": { + "mean": -0.002136333048618079, + "min": -0.018236353993415833, + "max": 5.7774828746914864e-05, + }, + "tp_min": {"mean": 3.998941094013752e-06, "min": 0.0, "max": 0.006604030728340149}, + "t2m_max": { + "mean": 292.271053668584, + "min": 224.75808715820312, + "max": 325.5970153808594, + }, + "pev_max": { + "mean": -9.391972474487043e-05, + "min": -0.004983663558959961, + "max": 0.02069767564535141, + }, + "slhf_max": {"mean": -1998.1564242578777, "min": -1504041.5, "max": 13011428.0}, + "ssr_max": {"mean": 14938754.622621803, "min": -0.0625, "max": 31967088.0}, + "sp_max": {"mean": 92268.74949649484, "min": 50269.9453125, "max": 106309.046875}, + "sshf_max": {"mean": 787876.1880326597, "min": -2240390.5, "max": 42488880.0}, + "e_max": { + "mean": -1.9515435042241873e-06, + "min": -0.0006014241371303797, + "max": 0.0046675666235387325, + }, + "tp_max": {"mean": 0.008162581792683693, "min": 0.0, "max": 0.397117555141449}, +} + + +class DeepExtremes2023Dataset(Dataset): + def __init__( + self, + folder: Union[Path, str], + metadata_files, + target: str, + variables=variables, + fp16=False, + is_lc_mask=True, + ): + if not isinstance(folder, Path): + folder = Path(folder) + self.metadata = sorted( + [ + ( + Path(folder, metadata_files["path"][idx][1:]), + metadata_files["start_date"][idx], + metadata_files["end_date"][idx], + ) + for idx in metadata_files.index + ] + ) # why sorted? + self.type = np.float16 if fp16 else np.float32 + self.target = target + self.variables = variables + self.is_lc_mask = is_lc_mask + + def __getitem__(self, idx: int) -> dict: + filepath, start_date, end_date = self.metadata[idx] + minicube = xr.open_dataset(filepath, engine="zarr").sel( + time=slice(start_date, end_date), + event_time=slice(start_date, min(end_date, pd.Timestamp(year=2021, month=12, day=31))), + ) + + if ( + minicube[self.variables["cloud_mask"]].time + != minicube[self.variables["s2_bands"]].B02.time + ).all(): + raise Exception( + "The first available imagery of sentinel-2 is not 4 + [5]" + + str(index_avail) + ) + + # Create the minicube + # s2 is 5 days, and already rescaled [0, 1] + s2_cube = ( + minicube[self.variables["s2_bands"]] + .to_array() + .values.transpose((1, 0, 2, 3)) + .astype(self.type) + ) # (time, channels, w, h) + + # s2_mask: + # 0 - free_sky + # 1 - cloud + # 2 - cloud_shadows + # 3 - snow + # 4 - masked_other_reasons + + s2_mask = ( + minicube[self.variables["cloud_mask"]] + .to_array() + .values.transpose((1, 0, 2, 3)) + .astype(self.type) + ) # (time, 1, w, h) + # mask = 1 where there is a cloud, 0 where data + s2_mask = s2_mask != 0 + + target = ( + self.target_computation(minicube).values[:, None, ...].astype(self.type) + ) + + # rescale all meteo variables? + for variable in self.variables["era5"]: + minicube[variable] = (minicube[variable] - statistic[variable]["min"]) / ( + statistic[variable]["max"] - statistic[variable]["min"] + ) + + # Era5land and Era5 dataset. Weather is 5-daily + meteo_cube = ( + minicube[self.variables["era5"]] + .to_array() + .values.transpose((1, 0)) + .astype(self.type) + ) + + topography = ( + minicube[self.variables["elevation"]].to_array().values.astype(self.type) + / 2000 + ) # c h w, rescaling + + # SCL is scene classification. i.e., it has a time dimension. + s2_scene_classification = ( + minicube[self.variables["s2_scene_classification"]] + .to_array() + .values.transpose((1, 0, 2, 3)) + .astype(self.type) + ) # c h w + + # NaN values handling + s2_cube = np.where(np.isnan(s2_cube), np.zeros(1).astype(self.type), s2_cube) + target = np.where(np.isnan(target), np.zeros(1).astype(self.type), target) + s2_mask = np.where( + np.isnan(s2_mask), np.ones(1).astype(self.type), s2_mask + ) # ?? s2_cube ? or s2_mask ? + topography = np.where( + np.isnan(topography), np.zeros(1).astype(self.type), topography + ) # ?? s2_cube ? or s2_mask + meteo_cube = np.where( + np.isnan(meteo_cube), np.zeros(1).astype(self.type), meteo_cube + ) # ?? s2_cube ? or s2_mask + + s2_scene_classification = np.where( + np.isnan(s2_scene_classification), + np.zeros(1).astype(self.type), + s2_scene_classification, + ) + + # lc mask = 1 where data + if self.is_lc_mask: + lc_mask = (s2_scene_classification == 4).astype( # 4 is vegetation pixel + self.type + ) + else: + # no lc mask + lc_mask = 1 + + # include scene classification in model? https://sentinels.copernicus.eu/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm-overview + # Concatenation + satellite_data = np.concatenate((target, s2_cube), axis=1) + + # Final minicube + data = { + "dynamic": [ + torch.from_numpy(satellite_data), + torch.from_numpy(meteo_cube), + ], + "dynamic_mask": [torch.from_numpy(s2_mask)], + "static": [torch.from_numpy(topography)], + "landcover": torch.from_numpy(s2_scene_classification), + "landcover_mask": torch.from_numpy(lc_mask).bool(), + "filepath": str(filepath), + "cubename": self.__name_getter(filepath, start_date), + } + return data + + def __len__(self) -> int: + return len(self.metadata) + + def __name_getter(self, path: Path, start_date) -> str: + """Helper function gets Cubename from a Path + + Args: + path (Path): One of Path/to/cubename.npz and Path/to/experiment_cubename.npz + + Returns: + [str]: cubename (has format tile_stuff.npz) + """ + components = path.name.split("/") + + components1 = components[-1].split("_") + name = "_".join(components1[0:3]) + output = name + "_" + str(start_date)[0:10] + return output + + def target_computation(self, minicube) -> str: + """Compute the vegetation index (VI) target""" + if self.target == "ndvi": + targ = (minicube.B8A - minicube.B04) / (minicube.B8A + minicube.B04 + 1e-6) + + if ( + self.target == "kndvi" + ): # TODO the denominator is not optimal, needs to be improved accordingly to the original paper + targ = np.tanh( + ((minicube.B08 - minicube.B04) / (minicube.BO8 + minicube.B04 + 1e-6)) + ** 2 + ) / np.tanh(1) + + if self.target == "anomalie_ndvi": + targ = ( + (minicube.B8A - minicube.B04) / (minicube.B8A + minicube.B04 + 1e-6) + ) - minicube.msc + + return targ + + +class DeepExtremes2023DataModule(pl.LightningDataModule): + def __init__(self, hparams: argparse.Namespace): + super().__init__() + self.save_hyperparameters(copy.deepcopy(hparams)) + self.base_dir = Path(hparams.base_dir) + + @staticmethod + def add_data_specific_args( + parent_parser: Optional[Union[argparse.ArgumentParser, list]] = None + ): + if parent_parser is None: + parent_parser = [] + elif not isinstance(parent_parser, list): + parent_parser = [parent_parser] + + parser = argparse.ArgumentParser(parents=parent_parser, add_help=False) + + parser.add_argument("--base_dir", type=str, default="data/datasets/") + parser.add_argument("--fold_path", type=str, default="mc_earthnet.csv") + parser.add_argument("--test_fold", type=int, default=10) + parser.add_argument("--val_fold", type=int, default=9) + + parser.add_argument("--continent_split", type=bool, default=False) + parser.add_argument("--test_track", type=str, default="iid") + parser.add_argument("--target", type=str, default="ndvi") + + parser.add_argument("--context_length", type=int, default=72) + parser.add_argument("--target_length", type=int, default=18) + + parser.add_argument("--is_lc_mask", type=bool, default=True) + + parser.add_argument("--fp16", type=str2bool, default=False) + + parser.add_argument("--train_batch_size", type=int, default=1) + parser.add_argument("--val_batch_size", type=int, default=1) + parser.add_argument("--test_batch_size", type=int, default=1) + + parser.add_argument("--val_split_seed", type=int, default=42) + + parser.add_argument( + "--num_workers", type=int, default=multiprocessing.cpu_count() + ) + + return parser + + def setup(self, stage: str = None): + if self.hparams.continent_split: + ( + train_subset, + val_subset, + spatial_test_subset, + temporal_test_subset, + continent_test_subset, + ) = self.dataset_split_continent() + else: + ( + train_subset, + val_subset, + spatial_test_subset, + temporal_test_subset, + ) = self.dataset_split() + + if stage == "fit" or stage is None: + self.earthnet_train = DeepExtremes2023Dataset( + self.base_dir, + train_subset, + target=self.hparams.target, + fp16=self.hparams.fp16, + is_lc_mask=self.hparams.is_lc_mask, + ) + + self.earthnet_val = DeepExtremes2023Dataset( + self.base_dir, + val_subset, + target=self.hparams.target, + fp16=self.hparams.fp16, + is_lc_mask=self.hparams.is_lc_mask, + ) + + if stage == "test" or stage is None: + if self.hparams.test_track == "iid": + self.earthnet_test = DeepExtremes2023Dataset( + self.base_dir, + spatial_test_subset, + target=self.hparams.target, + fp16=self.hparams.fp16, + is_lc_mask=self.hparams.lc_mask, + ) + elif self.hparams.test_track == "temporal": + self.earthnet_test = DeepExtremes2023Dataset( + self.base_dir, + temporal_test_subset, + target=self.hparams.target, + fp16=self.hparams.fp16, + is_lc_mask=self.hparams.lc_mask, + ) + + elif self.hparams.continent_split & ( + self.hparams.test_track == "continent" + ): + self.earthnet_test = DeepExtremes2023Dataset( + self.base_dir, + continent_test_subset, + target=self.hparams.target, + fp16=self.hparams.fp16, + is_lc_mask=self.hparams.lc_mask, + ) + + def train_dataloader(self) -> DataLoader: + return DataLoader( + self.earthnet_train, + batch_size=self.hparams.train_batch_size, + num_workers=self.hparams.num_workers, + pin_memory=True, + drop_last=True, + shuffle=True, + ) + + def val_dataloader(self) -> DataLoader: + return DataLoader( + self.earthnet_val, + batch_size=self.hparams.val_batch_size, + num_workers=self.hparams.num_workers, + pin_memory=True, + ) + + def test_dataloader(self) -> DataLoader: + return DataLoader( + self.earthnet_test, + batch_size=self.hparams.test_batch_size, + num_workers=self.hparams.num_workers, + pin_memory=True, + ) + + def dataset_split(self): + test_fold = self.hparams.test_fold + val_fold = self.hparams.val_fold + + # load csv + df = pd.read_csv(self.hparams.fold_path, delimiter=",")[ + ["path", "group", "check", "start_date"] + ] + + # 3 training minicubes between 2017 - 2020 (randomly starting in the first year), followed by 3 test minicubes in 2021 + df["start_date0"] = pd.to_datetime( + df["start_date"], format="%Y-%m-%dT%H:%M:%S.%f" + ) + df = df.drop("start_date", axis=1) + df["start_date2"] = df["start_date0"] + datetime.timedelta(days=(self.hparams.target_length + self.hparams.context_length)*5) + df["start_date3"] = df["start_date2"] + datetime.timedelta(days=(self.hparams.target_length + self.hparams.context_length)*5) + df["start_test1"] = df["start_date3"] + datetime.timedelta(days=(self.hparams.target_length + self.hparams.context_length)*5) + df["start_test2"] = df["start_test1"] + datetime.timedelta(days=self.hparams.target_length * 5) + df["start_test3"] = df["start_test2"] + datetime.timedelta(days=self.hparams.target_length * 5) + df = df.melt(["path", "group", "check"], value_name="start_date") + df["end_date"] = df["start_date"] + datetime.timedelta(days=(self.hparams.target_length + self.hparams.context_length) * 5 - 1 ) + + # temporal test set 2021 + temporal_test_subset = df.loc[ + (df["variable"].str.startswith("start_test")) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ].sort_values(by=["path","start_date"]) + # print(temporal_test_subset.iloc[:10]) + + # folds 2017 - 2020 + df = df.loc[df["variable"].str.startswith("start_date")].drop("variable", axis=1) + # print(df) + # print(df.loc[(df["check"] == 0) & (df["group"] >= 9) , :]) + + # training set + train_subset = df.loc[ + (df["group"] != test_fold) & (df["group"] != val_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ].sort_values(by=["path","start_date"]) + # print(train_subset.iloc[:10]) + + # validation set + val_subset = df.loc[ + (df["group"] == val_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ].sort_values(by=["path","start_date"]) + # print(val_subset.iloc[:10]) + + # iid test set + spatial_test_subset = df.loc[ + (df["group"] == test_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ].sort_values(by=["path","start_date"]) + # print(spatial_test_subset.iloc[:10]) + + return train_subset, val_subset, spatial_test_subset, temporal_test_subset + + def dataset_split_continent(self): + test_fold = self.hparams.test_fold + val_fold = self.hparams.val_fold + + # Read your CSV file into a pandas DataFrame + df = pd.read_csv(self.hparams.fold_path, delimiter=",")[ + [ + "path", + "group", + "check", + "start_date", + "lat", + "lon", + ] + ] + + # 3 training minicubes between 2017 - 2020 (randomly starting in the first year), followed by 3 test minicubes in 2021 + df["start_date"] = pd.to_datetime( + df["start_date"], format="%Y-%m-%dT%H:%M:%S.%f" + ) + df["start_date2"] = df["start_date"] + datetime.timedelta(days=450) + df["start_date3"] = df["start_date2"] + datetime.timedelta(days=450) + df["start_test1"] = df["start_date3"] + datetime.timedelta(days=450) + df["start_test2"] = df["start_test1"] + datetime.timedelta(days=90) + df["start_test3"] = df["start_test2"] + datetime.timedelta(days=90) + df = df.melt(["path", "group", "check", "lat", "lon"], value_name="start_date") + df["end_date"] = df["start_date"] + datetime.timedelta(days=449) + + # Check if the point is in Africa + def is_in_africa(row, africa): + # Create a Shapely Point from latitude and longitude + point = Point(row["lon"], row["lat"]) + + # Check if the point is in Africa + return africa.contains(point).any() + + # Load a world map dataset (included in geopandas) + world = gpd.read_file("https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_110m_admin_0_countries.geojson") + africa = world[world["continent"] == "Africa"] + # Apply the is_in_africa function to each row + df["in_africa"] = df.apply(lambda row: is_in_africa(row, africa), axis=1) + + # temporal test set 2021 + temporal_test_subset = df.loc[ + (df["variable"].str.startswith("start_test")), + ["path", "start_date", "end_date"], + ] + + # folds 2017 - 2020 + df = df.loc[df["variable"].str.startswith("start_date")].drop("variable", 1) + + # training set + train_subset = df.loc[ + (df["group"] != test_fold) + & (df["group"] != val_fold) + & (df["check"] == 0) + & (df["in_africa"] == False), + ["path", "start_date", "end_date"], + ] + + # validation set + val_subset = df.loc[ + (df["group"] == val_fold) & (df["check"] == 0) & (df["in_africa"] == False), + ["path", "start_date", "end_date"], + ] + + # iid test set + spatial_test_subset = df.loc[ + (df["group"] == test_fold) + & (df["check"] == 0) + & (df["in_africa"] == False), + ["path", "start_date", "end_date"], + ] + + africa_test_subset = df.loc[ + (df["check"] == 0) & (df["in_africa"] == True), + ["path", "start_date", "end_date"], + ] + + return ( + train_subset, + val_subset, + spatial_test_subset, + temporal_test_subset, + africa_test_subset, + ) diff --git a/earthnet_models_pytorch/data/en21_data.py b/earthnet_models_pytorch/data/en21_data.py index 36d4a91..85012b3 100755 --- a/earthnet_models_pytorch/data/en21_data.py +++ b/earthnet_models_pytorch/data/en21_data.py @@ -103,7 +103,6 @@ def __getitem__(self, idx: int) -> dict: if not self.use_meso_static_as_dynamic else [torch.from_numpy(highresstatic)] ), - "static_mask": [], "filepath": str(filepath), "cubename": self.__name_getter(filepath), } diff --git a/earthnet_models_pytorch/data/en21x_data.py b/earthnet_models_pytorch/data/en21x_data.py index 122a426..328f7d1 100755 --- a/earthnet_models_pytorch/data/en21x_data.py +++ b/earthnet_models_pytorch/data/en21x_data.py @@ -18,7 +18,6 @@ class EarthNet2021XDataset(Dataset): - def __init__( self, folder: Union[Path, str], @@ -30,6 +29,8 @@ def __init__( start_month_extreme=None, dl_cloudmask=False, allow_fastaccess=False, + lc_min = 40, + lc_max = 90, ): if not isinstance(folder, Path): folder = Path(folder) @@ -64,6 +65,8 @@ def __init__( self.s2_bands = s2_bands self.eobs_vars = eobs_vars + self.lc_min = lc_min + self.lc_max = lc_max self.eobs_agg = eobs_agg self.static_vars = static_vars self.start_month_extreme = start_month_extreme @@ -134,11 +137,9 @@ def __init__( print(f"dataset has {len(self)} samples") def __getitem__(self, idx: int) -> dict: - filepath = self.filepaths[idx] if self.fast_access: - npz = np.load(filepath) data = { "dynamic": [ @@ -244,6 +245,7 @@ def __getitem__(self, idx: int) -> dict: .transpose("variable", "lat", "lon") .values ) # c h w + lc_mask = (lc <= self.lc_min) | (lc >= self.lc_max) data = { "dynamic": [ @@ -252,8 +254,8 @@ def __getitem__(self, idx: int) -> dict: ], "dynamic_mask": [torch.from_numpy(sen2mask.astype(self.type))], "static": [torch.from_numpy(staticarr.astype(self.type))], - "static_mask": [], "landcover": torch.from_numpy(lc.astype(self.type)), + "landcover_mask": torch.from_numpy(lc_mask).bool(), "filepath": str(filepath), "cubename": filepath.stem, } @@ -305,7 +307,6 @@ def add_data_specific_args( def setup(self, stage: str = None): if stage == "fit" or stage is None: - if self.hparams.new_valset: self.earthnet_train = EarthNet2021XDataset( self.base_dir / "train", diff --git a/earthnet_models_pytorch/data/en21x_data_old.py b/earthnet_models_pytorch/data/en21x_data_old.py index 48b830e..e14813a 100755 --- a/earthnet_models_pytorch/data/en21x_data_old.py +++ b/earthnet_models_pytorch/data/en21x_data_old.py @@ -15,11 +15,12 @@ class EarthNet2021XOldDataset(Dataset): - def __init__( self, folder: Union[Path, str], fp16=False, + lc_min=40, + lc_max=90, spatial_eobs=True, eobs_spread=False, soilgrids_all=False, @@ -33,6 +34,9 @@ def __init__( self.type = np.float16 if fp16 else np.float32 + self.lc_min = lc_min + self.lc_max = lc_max + self.spatial_eobs = spatial_eobs self.eobs_spread = eobs_spread @@ -136,7 +140,6 @@ def __init__( ] def __getitem__(self, idx: int) -> dict: - filepath = self.filepaths[idx] minicube = xr.open_dataset(filepath) @@ -220,12 +223,15 @@ def __getitem__(self, idx: int) -> dict: lc[np.isnan(lc)] = 0 + lc_mask = (lc <= self.lc_min) | (lc >= self.lc_max) + data = { "dynamic": [torch.from_numpy(kndvi), torch.from_numpy(eobs)], "dynamic_mask": [], "static": [torch.from_numpy(highresstatic)], "static_mask": [], "landcover": torch.from_numpy(lc), + "landcover_mask": torch.from_numpy(lc_mask).bool(), "filepath": str(filepath), "cubename": self.__name_getter(filepath), } @@ -253,7 +259,6 @@ def __name_getter(self, path: Path) -> str: class EarthNet2021XpxOldDataset(Dataset): - def __init__( self, folder: Union[Path, str], @@ -373,7 +378,6 @@ def __init__( print("Initialized dataset") def __getitem__(self, idx: int) -> dict: - pixel = self.dataset.isel(loc=idx) kndvi = pixel.kndvi.values[:, None].astype(self.type) # t c @@ -445,6 +449,7 @@ def __getitem__(self, idx: int) -> dict: lc = pixel.lc.values[None, ...].astype(self.type) # c lc[np.isnan(lc)] = 0 + lc_mask = (lc <= self.lc_min) | (lc >= self.lc_max) data = { "dynamic": [torch.from_numpy(kndvi), torch.from_numpy(eobs)], @@ -452,6 +457,7 @@ def __getitem__(self, idx: int) -> dict: "static": [torch.from_numpy(highresstatic)], "static_mask": [], "landcover": torch.from_numpy(lc), + "landcover_mask": torch.from_numpy(lc_mask).bool(), "filepath": "", "cubename": str(pixel["loc"].values), } @@ -463,7 +469,6 @@ def __len__(self) -> int: class EarthNet2021XOldDataModule(pl.LightningDataModule): - def __init__(self, hparams: argparse.Namespace): super().__init__() self.save_hyperparameters(copy.deepcopy(hparams)) @@ -484,6 +489,8 @@ def add_data_specific_args( parser.add_argument("--test_track", type=str, default="iid") parser.add_argument("--fp16", type=str2bool, default=False) + parser.add_argument("--lc_min", type=int, default=40) + parser.add_argument("--lc_max", type=int, default=90) parser.add_argument("--spatial_eobs", type=str2bool, default=True) parser.add_argument("--eobs_spread", type=str2bool, default=True) parser.add_argument("--soilgrids_all", type=str2bool, default=True) @@ -505,6 +512,8 @@ def setup(self, stage: str = None): earthnet_corpus = EarthNet2021XOldDataset( self.base_dir / "train", fp16=self.hparams.fp16, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, spatial_eobs=self.hparams.spatial_eobs, eobs_spread=self.hparams.eobs_spread, soilgrids_all=self.hparams.soilgrids_all, @@ -530,6 +539,8 @@ def setup(self, stage: str = None): self.earthnet_test = EarthNet2021XOldDataset( self.base_dir / "test" / self.hparams.test_track, fp16=self.hparams.fp16, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, spatial_eobs=self.hparams.spatial_eobs, eobs_spread=self.hparams.eobs_spread, soilgrids_all=self.hparams.soilgrids_all, @@ -562,7 +573,6 @@ def test_dataloader(self) -> DataLoader: class EarthNet2021XpxOldDataModule(pl.LightningDataModule): - def __init__(self, hparams: argparse.Namespace): super().__init__() self.save_hyperparameters(copy.deepcopy(hparams)) @@ -583,6 +593,9 @@ def add_data_specific_args( parser.add_argument("--test_track", type=str, default="iid") parser.add_argument("--fp16", type=str2bool, default=False) + parser.add_argument("--lc_min", type=int, default=40) + parser.add_argument("--lc_max", type=int, default=90) + parser.add_argument("--spatial_eobs", type=str2bool, default=True) parser.add_argument("--eobs_spread", type=str2bool, default=True) parser.add_argument("--soilgrids_all", type=str2bool, default=True) @@ -605,6 +618,8 @@ def setup(self, stage: str = None): earthnet_corpus = EarthNet2021XOldDataset( self.base_dir / "train", fp16=self.hparams.fp16, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, spatial_eobs=self.hparams.spatial_eobs, eobs_spread=self.hparams.eobs_spread, soilgrids_all=self.hparams.soilgrids_all, @@ -629,6 +644,8 @@ def setup(self, stage: str = None): self.earthnet_train = EarthNet2021XpxOldDataset( self.base_dir, fp16=self.hparams.fp16, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, eobs_spread=self.hparams.eobs_spread, soilgrids_all=self.hparams.soilgrids_all, ) diff --git a/earthnet_models_pytorch/data/en22_data.py b/earthnet_models_pytorch/data/en22_data.py index 170d517..4e91e32 100755 --- a/earthnet_models_pytorch/data/en22_data.py +++ b/earthnet_models_pytorch/data/en22_data.py @@ -157,19 +157,21 @@ def __getitem__(self, idx: int) -> dict: meteo[np.isnan(meteo)] = 0 - highresstatic = minicube.dem.values[None, ...].astype(self.type) # c h w - highresstatic /= 2000 + highresstatic = minicube.dem.values[None, ...].astype(self.type) / 2000 highresstatic[np.isnan(highresstatic)] = 0 lc = minicube.lc.values[None, ...].astype(self.type) # c h w lc[np.isnan(lc)] = 0 + lc_mask = (lc >= self.lc_min) & (lc <= self.lc_max) + data = { "dynamic": [torch.from_numpy(hr), self.transform_meteo(meteo)], "dynamic_mask": [], "static": [torch.from_numpy(highresstatic)], "static_mask": [], "landcover": torch.from_numpy(lc), + "landcover_mask": torch.from_numpy(lc_mask).bool(), "filepath": str(filepath), "cubename": self.__name_getter(filepath), } diff --git a/earthnet_models_pytorch/data/en23_data.py b/earthnet_models_pytorch/data/en23_data.py index 266c2af..2c7559a 100755 --- a/earthnet_models_pytorch/data/en23_data.py +++ b/earthnet_models_pytorch/data/en23_data.py @@ -340,7 +340,13 @@ class EarthNet2023Dataset(Dataset): def __init__( - self, folder: Union[Path, str], target: str, variables=variables, fp16=False + self, + folder: Union[Path, str], + target: str, + lc_min: int, + lc_max: int, + variables=variables, + fp16=False, ): if not isinstance(folder, Path): folder = Path(folder) @@ -349,9 +355,11 @@ def __init__( if len(sorted(list(folder.glob("*.nc")))) > 0 else sorted(list(folder.glob("*/*.nc"))) ) - print(folder) + print("path of the dataset: ", folder) self.type = np.float16 if fp16 else np.float32 self.target = target + self.lc_min = lc_min + self.lc_max = lc_max self.variables = variables def __getitem__(self, idx: int) -> dict: @@ -406,7 +414,6 @@ def __getitem__(self, idx: int) -> dict: ) # c h w # Rescaling - for variable in ( self.variables["era5"] + self.variables["era5lands"] @@ -443,6 +450,7 @@ def __getitem__(self, idx: int) -> dict: landcover = np.where( np.isnan(landcover), np.zeros(1).astype(self.type), landcover ) + lc_mask = (landcover <= self.lc_min) | (landcover >= self.lc_max) # Concatenation satellite_data = np.concatenate((target, s2_cube), axis=1) @@ -457,6 +465,7 @@ def __getitem__(self, idx: int) -> dict: "static": [torch.from_numpy(topography), torch.from_numpy(sg_cube)], # "target": torch.from_numpy(target), "landcover": torch.from_numpy(landcover), + "landcover_mask": torch.from_numpy(lc_mask).bool(), "filepath": str(filepath), "cubename": self.__name_getter(filepath), } @@ -530,6 +539,9 @@ def add_data_specific_args( parser.add_argument("--test_track", type=str, default="iid") parser.add_argument("--target", type=str, default="ndvi") + parser.add_argument("--lc_min", type=int, default=40) + parser.add_argument("--lc_max", type=int, default=90) + parser.add_argument("--fp16", type=str2bool, default=False) parser.add_argument("--train_batch_size", type=int, default=1) @@ -549,6 +561,8 @@ def setup(self, stage: str = None): earthnet_full = EarthNet2023Dataset( self.base_dir / "train", target=self.hparams.target, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, fp16=self.hparams.fp16, ) self.earthnet_train, self.earthnet_val = random_split( @@ -561,6 +575,8 @@ def setup(self, stage: str = None): self.earthnet_test = EarthNet2023Dataset( self.base_dir / "test", target=self.hparams.target, + lc_min=self.hparams.lc_min, + lc_max=self.hparams.lc_max, fp16=self.hparams.fp16, ) diff --git a/earthnet_models_pytorch/metric/__init__.py b/earthnet_models_pytorch/metric/__init__.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/metric/nnse.py b/earthnet_models_pytorch/metric/nnse.py index 3aab658..1c69e53 100755 --- a/earthnet_models_pytorch/metric/nnse.py +++ b/earthnet_models_pytorch/metric/nnse.py @@ -27,8 +27,6 @@ def __init__( self.add_state("nnse_sum", default=torch.tensor(0.0), dist_reduce_fx="sum") self.add_state("n_obs", default=torch.tensor(1e-6), dist_reduce_fx="sum") - self.lc_min = lc_min - self.lc_max = lc_max self.ndvi_pred_idx = ndvi_pred_idx self.ndvi_targ_idx = ndvi_targ_idx self.mask_hq_only = mask_hq_only @@ -85,6 +83,14 @@ def update(self, preds, batch): nse = 1 - sum_squared_error / (sum_squared_deviation + 1e-8) # b c h w + # Landcover mask + lc_mask = batch["landcover_mask"] + + if len(lc_mask.shape) < 5: # spatial landcover mask + lc_mask = lc_mask.unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) + else: # spato-temporal landcover mask + lc_mask = lc_mask[:, self.context_length : self.context_length + self.target_length, ...] + if self.mask_hq_only: lc_mask = ( (lc >= self.lc_min).bool() diff --git a/earthnet_models_pytorch/metric/rmse.py b/earthnet_models_pytorch/metric/rmse.py index bde47ea..caee72f 100755 --- a/earthnet_models_pytorch/metric/rmse.py +++ b/earthnet_models_pytorch/metric/rmse.py @@ -6,8 +6,6 @@ class RootMeanSquaredError(Metric): # Each state variable should be called using self.add_state(...) def __init__( self, - lc_min: int, - lc_max: int, context_length: int, target_length: int, dist_sync_on_step: bool = False, @@ -42,20 +40,14 @@ def __init__( ) # State variables for the metric - self.add_state( - "sum_squared_error", default=torch.tensor(0.0), dist_reduce_fx="sum" - ) - self.add_state("total", default=torch.tensor(0), dist_reduce_fx="sum") + self.add_state("total", default=torch.tensor(0.0), dist_reduce_fx="sum") + self.add_state("n_samples", default=torch.tensor(0), dist_reduce_fx="sum") self.add_state("current_scores", default=torch.tensor(0), dist_reduce_fx=None) self.context_length = context_length self.target_length = target_length - self.lc_min = lc_min - self.lc_max = lc_max - print( - f"Using Masked RootMeanSquaredError metric Loss with Landcover boundaries ({self.lc_min, self.lc_max})." - ) + print(f"Using Masked RootMeanSquaredError metric Loss.") def update(self, preds, targs): """ @@ -90,13 +82,14 @@ def update(self, preds, targs): ) # Landcover mask - lc = targs["landcover"] - lc_mask = ( - ((lc <= self.lc_min).bool() | (lc >= self.lc_max).bool()) - .type_as(s2_mask) - .unsqueeze(1) - .repeat(1, preds.shape[1], 1, 1, 1) - ) + lc_mask = targs["landcover_mask"] + + if len(lc_mask.shape) < 5: # spatial landcover mask + lc_mask = lc_mask.unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) + else: # spato-temporal landcover mask + lc_mask = lc_mask[ + :, self.context_length : self.context_length + self.target_length, ... + ] mask = s2_mask * lc_mask @@ -105,9 +98,10 @@ def update(self, preds, targs): n_obs = (mask == 1).sum((1, 2, 3, 4)) # sum of pixel with vegetation # Update the states variables - self.current_scores = sum_squared_error / (n_obs + 1e-8) - self.sum_squared_error += sum_squared_error.sum() - self.total += n_obs.sum() + self.current_scores = torch.sqrt(sum_squared_error / (n_obs + 1e-8)) + self.total += (sum_squared_error / (n_obs + 1e-8)).sum() + self.n_samples += len(n_obs.shape) + # print(n_obs) def compute(self): """ @@ -118,7 +112,8 @@ def compute(self): dict Dictionary containing the computed RMSE for vegetation pixels. """ - return {"RMSE_Veg": torch.sqrt(self.sum_squared_error / self.total)} + # print(self.total, self.n_samples, torch.sqrt(self.total / (self.n_samples))) + return {"RMSE_Veg": torch.sqrt(self.total / (self.n_samples))} def compute_batch(self, targs): """ diff --git a/earthnet_models_pytorch/model/conditioning_layer.py b/earthnet_models_pytorch/model/conditioning_layer.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/model/contextformer.py b/earthnet_models_pytorch/model/contextformer.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/model/convlstm_ae.py b/earthnet_models_pytorch/model/convlstm_ae.py index 0750159..5b2e7bc 100755 --- a/earthnet_models_pytorch/model/convlstm_ae.py +++ b/earthnet_models_pytorch/model/convlstm_ae.py @@ -1,5 +1,5 @@ """ConvLSTM_ae - ConvLSTM with an encoding-decoding architecture +ConvLSTM with an encoding-decoding architecture """ import argparse @@ -168,7 +168,7 @@ def __init__(self, hparams: argparse.Namespace): self.decoder_1_convlstm = ConvLSTMCell( input_dim=( - self.hparams.num_inputs - 4 + self.hparams.num_inputs - self.hparams.num_s2bands if self.hparams.decoder_input_subtract_s2bands else self.hparams.num_inputs ), # nb of s2 bands. @@ -203,7 +203,7 @@ def __init__(self, hparams: argparse.Namespace): @staticmethod def add_model_specific_args( - parent_parser: Optional[Union[argparse.ArgumentParser, list]] = None + parent_parser: Optional[Union[argparse.ArgumentParser, list]] = None, ): """ Add model-specific arguments to the command-line argument parser. @@ -245,6 +245,7 @@ def add_model_specific_args( parser.add_argument( "--decoder_input_subtract_s2bands", type=str2bool, default=True ) + parser.add_argument("--num_s2bands", type=int, default=7) parser.add_argument("--weather_is_aggregated", type=str2bool, default=False) return parser @@ -320,7 +321,7 @@ def forward( # static = torch.cat((data["static"][0], data["static"][1]), dim=1) # static = torch.cat((static, landcover), dim=1) - static = data["static"][0][:, :3, ...] + static = data["static"][0] # Get the dimensions of the input data. Shape: batch size, temporal size, number of channels, height, width b, t, c, h, w = sentinel.shape @@ -374,6 +375,8 @@ def forward( if self.hparams.use_weather: if self.hparams.weather_is_aggregated: weather_t = weather[:, t, ...].expand(-1, -1, h, w) + elif weather.shape[1] == 90: # weather is 5 days resolution + weather_t = weather[:, t, ...].repeat(1, 1, 128, 128) else: weather_t = ( weather[:, t : t + 5, ...] @@ -426,6 +429,10 @@ def forward( # Prepare input for decoder ConvLSTM cells if self.hparams.weather_is_aggregated: weather_t = weather[:, context_length + t, ...].expand(-1, -1, h, w) + elif weather.shape[1] == 90: # weather is 5 days resolution + weather_t = weather[:, context_length + t, ...].repeat( + 1, 1, 128, 128 + ) else: weather_t = ( weather[:, context_length + t : context_length + t + 5, ...] diff --git a/earthnet_models_pytorch/model/layer_utils.py b/earthnet_models_pytorch/model/layer_utils.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/model/nextframe_unet.py b/earthnet_models_pytorch/model/nextframe_unet.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/model/spatiotemporal_unet.py b/earthnet_models_pytorch/model/spatiotemporal_unet.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/model/unet_lstm.py b/earthnet_models_pytorch/model/unet_lstm.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/task/__init__.py b/earthnet_models_pytorch/task/__init__.py old mode 100644 new mode 100755 diff --git a/earthnet_models_pytorch/task/loss.py b/earthnet_models_pytorch/task/loss.py index 6d9b04f..d29cd53 100755 --- a/earthnet_models_pytorch/task/loss.py +++ b/earthnet_models_pytorch/task/loss.py @@ -98,8 +98,6 @@ def forward(self, preds, batch, aux, current_step=None): class MaskedPixelwiseLoss(nn.Module): def __init__( self, - lc_min=None, - lc_max=None, context_length=None, target_length=None, ndvi_pred_idx=0, @@ -113,15 +111,6 @@ def __init__( ): super().__init__() - self.lc_min = ( - lc_min if lc_min else None - ) # landcover boudaries of vegetation (to select only pixel with vegetation) - self.lc_max = lc_max if lc_max else None - self.use_lc = lc_min & lc_max - if not self.use_lc: - print( - f"WARNING. The boundaries of the landcover map are not definite. Loss calculated on all pixels including non-vegetation pixels." - ) self.context_length = context_length self.target_length = target_length self.ndvi_pred_idx = ndvi_pred_idx # index of the NDVI band @@ -130,16 +119,17 @@ def __init__( self.scale_by_std = scale_by_std if self.scale_by_std: print( - f"Using Masked L2/Std NDVI Loss with Landcover boundaries ({self.lc_min, self.lc_max})." + f"Using Masked L2/Std NDVI Loss." ) else: print( - f"Using Masked L2 NDVI Loss with Landcover boundaries ({self.lc_min, self.lc_max})." + f"Using Masked L2 NDVI Loss." ) self.extra_aux_loss_term = extra_aux_loss_term self.extra_aux_loss_weight = extra_aux_loss_weight self.setting = setting + self.is_lc_mask = True if 'is_lc_mask' not in list(kwargs.keys()) else kwargs['is_lc_mask'] def forward(self, preds, batch, aux, current_step=None): logs = {} @@ -165,17 +155,20 @@ def forward(self, preds, batch, aux, current_step=None): .type_as(preds) ) - # Landcover mask - lc = batch["landcover"] - if self.setting == "en23": - lc_bool = (lc <= self.lc_min).bool() | (lc >= self.lc_max).bool() + # Mask certain land covers + if self.is_lc_mask: + # Landcover mask + lc_mask = batch["landcover_mask"] + if len(lc_mask.shape) < 5: # spatial landcover mask + lc_mask = lc_mask.unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) + else: # spato-temporal landcover mask + lc_mask = lc_mask[:, self.context_length : self.context_length + self.target_length, ...] + + # mask should be = 1 where there are data + mask = s2_mask * lc_mask else: - lc_mask = (lc >= self.lc_min).bool() & (lc <= self.lc_max).bool() + mask = s2_mask - lc_mask = ( - lc_bool.type_as(s2_mask).unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) - ) - mask = s2_mask * lc_mask # MSE computation sum_squared_error = torch.pow((preds - targets) * mask, 2).sum() @@ -188,8 +181,6 @@ def forward(self, preds, batch, aux, current_step=None): class MaskedL2NDVILoss(nn.Module): def __init__( self, - lc_min=None, - lc_max=None, context_length=None, target_length=None, ndvi_pred_idx=0, @@ -203,16 +194,6 @@ def __init__( **kwargs, ): super().__init__() - - self.lc_min = ( - lc_min if lc_min else None - ) # landcover boudaries of vegetation (to select only pixel with vegetation) - self.lc_max = lc_max if lc_max else None - self.use_lc = lc_min & lc_max - if not self.use_lc: - print( - f"WARNING. The boundaries of the landcover map are not definite. Loss calculated on all pixels including non-vegetation pixels." - ) self.context_length = context_length self.target_length = target_length self.ndvi_pred_idx = ndvi_pred_idx # index of the NDVI band @@ -222,11 +203,11 @@ def __init__( self.weight_by_std = weight_by_std if self.scale_by_std: print( - f"Using Masked L2/Std NDVI Loss with Landcover boundaries ({self.lc_min, self.lc_max})." + f"Using Masked L2/Std NDVI Loss." ) else: print( - f"Using Masked L2 NDVI Loss with Landcover boundaries ({self.lc_min, self.lc_max})." + f"Using Masked L2 NDVI Loss." ) self.extra_aux_loss_term = extra_aux_loss_term @@ -250,10 +231,13 @@ def forward(self, preds, batch, aux, current_step=None): ) # b t c h w # Landcover mask - lc = batch["landcover"] - lc_mask = ((lc >= self.lc_min).bool() & (lc <= self.lc_max).bool()).type_as( - preds - ) # b c h w + # Landcover mask + lc_mask = batch["landcover_mask"] + if len(lc_mask.shape) < 5: # spatial landcover mask + lc_mask = lc_mask.unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) + else: # spato-temporal landcover mask + lc_mask = lc_mask[:, self.context_length : self.context_length + self.target_length, ...] + ndvi_targ = batch["dynamic"][0][ :, self.context_length : self.context_length + self.target_length, diff --git a/earthnet_models_pytorch/task/workflow.py b/earthnet_models_pytorch/task/workflow.py index 3fc8f09..8d71ce8 100755 --- a/earthnet_models_pytorch/task/workflow.py +++ b/earthnet_models_pytorch/task/workflow.py @@ -40,8 +40,8 @@ def __init__(self, model: nn.Module, hparams: argparse.Namespace): self.context_length = hparams.context_length self.target_length = hparams.target_length - self.lc_min = hparams.lc_min - self.lc_max = hparams.lc_max + #self.lc_min = hparams.lc_min + #self.lc_max = hparams.lc_max self.n_stochastic_preds = hparams.n_stochastic_preds @@ -90,8 +90,8 @@ def add_task_specific_args( parser.add_argument("--target_length", type=int, default=20) # Landcover bounds. Used as mask on the non-vegetation pixel. - parser.add_argument("--lc_min", type=int, default=10) - parser.add_argument("--lc_max", type=int, default=20) + #parser.add_argument("--lc_min", type=int, default=10) + #parser.add_argument("--lc_max", type=int, default=20) # Number of stochastic prediction for statistical models. parser.add_argument("--n_stochastic_preds", type=int, default=1) @@ -295,7 +295,7 @@ def test_step(self, batch, batch_idx): # Targets targ_path = Path(batch["filepath"][j]) targ_cube = xr.open_dataset(targ_path) - + lat = targ_cube.lat lon = targ_cube.lon @@ -332,6 +332,59 @@ def test_step(self, batch, batch_idx): pred_cube.to_netcdf( pred_path, encoding={"ndvi_pred": {"dtype": "float32"}} ) + elif self.hparams.setting in ["de23"]: + # Targets + targ_path = Path(batch["filepath"][j]) + targ_cube = xr.open_dataset(targ_path, engine='zarr') + start_time = batch["cubename"][j][-10:] + t0 = np.datetime64(start_time) + t1 = t0 + np.timedelta64(self.context_length*5-1, 'D') + t2 = t1 + np.timedelta64(self.target_length*5-1, 'D') + lat = targ_cube.x + lon = targ_cube.y + ndvi_preds = preds[j, :, 0, ...].detach().cpu().numpy() + pred_cube = xr.Dataset( + { + "ndvi_pred": xr.DataArray( + data=ndvi_preds, + coords={ + "time": targ_cube.time.sel( + time=slice(t1, t2) + ), + "y": lon, + "x": lat, + }, + dims=["time", "y", "x"], + ) + } + ) + # print(pred_cube) + pred_dir = self.pred_dir + pred_path = pred_dir / Path(batch["cubename"][j] + ".zarr") # targ_path.parent.stem / targ_path.name + pred_path.parent.mkdir(parents=True, exist_ok=True) + + if pred_path.is_file(): + pred_path.unlink() + + if not pred_path.is_file(): + pred_cube.to_zarr( + pred_path, + chunk_store=None, + mode="w", + synchronizer=None, + group=None, + encoding={"ndvi_pred": {"dtype": "float32"}}, + compute=True, + consolidated=True, + append_dim=None, + region=None, + safe_chunks=True, + storage_options=None, + zarr_version=None + ) + # pred_cube.to_netcdf( + # pred_path, encoding={"ndvi_pred": {"dtype": "float32"}} + # ) elif self.hparams.setting in ["en21xold", "en22"]: # Targets @@ -468,7 +521,6 @@ def on_test_epoch_end(self): """Called at the end of a test epoch with the output of all test steps.""" if self.hparams.compute_metric_on_test: self.pred_dir.mkdir(parents=True, exist_ok=True) - print(self.pred_dir) with open( self.pred_dir / f"individual_scores_{self.global_rank}.json", "w" ) as fp: diff --git a/earthnet_models_pytorch/utils/logging.py b/earthnet_models_pytorch/utils/logging.py index 7a7364e..28c86b5 100755 --- a/earthnet_models_pytorch/utils/logging.py +++ b/earthnet_models_pytorch/utils/logging.py @@ -117,6 +117,12 @@ def log_viz( targs = batch["dynamic"][0] nrow = 9 if targs.shape[1] % 9 == 0 else 10 + if "landcover_mask" in batch: + # print("lc_mask in batch") + lc_mask = batch["landcover_mask"] + # else: + # print("need to set mask from lc") + if "landcover" in batch: lc = batch["landcover"] if setting == "en21x": @@ -192,6 +198,34 @@ def log_viz( setting=setting, ) grid = torchvision.utils.make_grid(ndvi, nrow=nrow) + elif setting == "de23": + # print(lc_mask.size()) + if len(lc_mask.shape) < 5: # spatial landcover mask + lc_mask1 = lc_mask.unsqueeze(1).repeat(1, preds.shape[1], 1, 1, 1) + else: # spato-temporal landcover mask + # print(preds.size()) + target_length = preds.size(dim=1) + context_length = lc_mask.size(dim=1) - target_length + lc_mask1 = lc_mask[ + j, context_length : context_length + target_length, ... + ] + # print(lc_mask1.size()) + ndvi = veg_colorize( + preds[j, ...].squeeze(), + mask=None if "landcover_mask" not in batch else lc_mask1, + setting=setting, + ) + text = f"Cube: {scores[j]['name']} Score: {scores[j]['rmse' if 'rmse' in scores[j] else 'veg_score']:.4f}" + grid = torchvision.utils.make_grid(ndvi, nrow=nrow) + text = ( + torch.tensor( + text_phantom(text, width=grid.shape[-1]), + dtype=torch.float32, + device=targs.device, + ) + .type_as(grid) + .permute(2, 0, 1) + ) else: ndvi = veg_colorize( @@ -296,6 +330,35 @@ def log_viz( clouds=masks[j, :, 0, ...] if masks is not None else None, mode="ndvi", ) + elif setting == "de23": + rgb = torch.cat( + [ + targs[j, :, 3, ...].unsqueeze(1) * 10000, + targs[j, :, 2, ...].unsqueeze(1) * 10000, + targs[j, :, 1, ...].unsqueeze(1) * 10000, + ], + dim=1, + ) + grid = torchvision.utils.make_grid( + rgb, nrow=nrow, normalize=True, value_range=(0, 5000) + ) + tensorboard_logger.add_image( + f"Cube: {batch_idx*preds.shape[0] + j} RGB Targets", + grid, + current_epoch, + ) + + ndvi = veg_colorize( + targs[j, :, 0, ...], + mask=( + None + if "landcover_mask" not in batch + else lc_mask[j, :, 0, ...] + ), + clouds=masks[j, :, 0, ...], + setting=setting, + ) + else: # kndvi ndvi = veg_colorize( targs[j, :, 0, ...], diff --git a/earthnet_models_pytorch/utils/parse.py b/earthnet_models_pytorch/utils/parse.py index 54033bb..cfce3b0 100755 --- a/earthnet_models_pytorch/utils/parse.py +++ b/earthnet_models_pytorch/utils/parse.py @@ -129,17 +129,19 @@ def parse_setting(setting_file, track = None): setting_dict["Task"]["val_batch_size"] = setting_dict["Data"]["val_batch_size"] setting_dict["Task"]["test_batch_size"] = setting_dict["Data"]["test_batch_size"] - setting_dict["Task"]["lc_min"] = setting_dict["Task"]["loss"]["lc_min"] - setting_dict["Task"]["lc_max"] = setting_dict["Task"]["loss"]["lc_max"] + # legacy for lc_min defined in Task config + if "lc_min" and "lc_max" in setting_dict["Task"]["loss"]: + setting_dict["Data"]["lc_min"] = setting_dict["Task"]["loss"]["lc_min"] + setting_dict["Data"]["lc_max"] = setting_dict["Task"]["loss"]["lc_max"] - setting_dict["Task"]["loss"]["setting"] = setting_dict["Setting"] - + del setting_dict["Task"]["loss"]["lc_min"] + del setting_dict["Task"]["loss"]["lc_max"] + + #setting_dict["Task"]["loss"]["setting"] = setting_dict["Setting"] if "metric_kwargs" not in setting_dict["Task"]: setting_dict["Task"]["metric_kwargs"] = {} setting_dict["Task"]["metric_kwargs"]["context_length"] = setting_dict["Task"]["context_length"] setting_dict["Task"]["metric_kwargs"]["target_length"] = setting_dict["Task"]["target_length"] - setting_dict["Task"]["metric_kwargs"]["lc_min"] = setting_dict["Task"]["lc_min"] - setting_dict["Task"]["metric_kwargs"]["lc_max"] = setting_dict["Task"]["lc_max"] if "model_shedules" in setting_dict["Task"]["metric_kwargs"]: setting_dict["Task"]["metric_kwargs"]["shedulers"] = setting_dict["Task"]["metric_kwargs"]["model_shedules"] diff --git a/requirements.txt b/requirements.txt old mode 100644 new mode 100755 index 3a0bf44..ff1528b --- a/requirements.txt +++ b/requirements.txt @@ -1,18 +1,20 @@ +dask==2023.11.0 earthnet==0.3.9 fire==0.4.0 -matplotlib==3.5.1 +matplotlib==3.6.2 numpy==1.23.5 -pandas==1.4.1 +pandas==1.3.5 Pillow==9.3.0 pytorch_lightning==1.8.5.post0 PyYAML==6.0 -scipy==1.11.2 +scipy==1.11.4 segmentation_models_pytorch==0.2.1 setuptools==59.5.0 +statsmodels==0.14.0 streamlit==1.17.0 timm==0.4.12 torch==1.13.1 torchmetrics==0.11.0 torchvision==0.14.1 -tqdm==4.66.1 -xarray==2022.12.0 +tqdm==4.64.1 +xarray==2022.12.0 \ No newline at end of file diff --git a/scripts/debug.py b/scripts/debug.py index 6701719..290146a 100755 --- a/scripts/debug.py +++ b/scripts/debug.py @@ -13,7 +13,6 @@ import sys import torch import pytorch_lightning as pl -from pytorch_lightning.strategies import DDPStrategy from earthnet_models_pytorch.model import MODELS from earthnet_models_pytorch.task import SpatioTemporalTask diff --git a/scripts/fastacces_data.py b/scripts/fastacces_data.py old mode 100644 new mode 100755 diff --git a/scripts/postprocessing/de23_metric_computation.py b/scripts/postprocessing/de23_metric_computation.py new file mode 100644 index 0000000..2ad8d44 --- /dev/null +++ b/scripts/postprocessing/de23_metric_computation.py @@ -0,0 +1,292 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import json +from tqdm import tqdm +import multiprocessing as mp +import os +from scipy.stats import pearsonr +from argparse import ArgumentParser + + +def calculate_variable_statistics(target, pred, anomalie=False, subset="all", cubename="", method = "by_frame", season = "DJF"): + """Function to calculate statistics for each variable in a sample""" + t0 = pred.time.values[0] + target_len = len(pred.time) + # time start index + it = np.where(target.time==t0)[0][0] + # first target_len steps, every 5 days + tind = range(it,it + target_len, 1) + # event_time start index + ie = np.where(target.event_time == t0-np.timedelta64(12,'h'))[0][0] + # every day + eind = range(ie, ie + 5*target_len, 1) + + targ = (target.B8A - target.B04) / (target.B8A + target.B04 + 1e-8)[ + tind, ... + ] + if anomalie: + targ = targ - target.msc[tind, ...] + + pred = pred.ndvi_pred + + # if any day in the test period is extreme, tag prediction as extreme (could be adapted by type of extreme) + extremes = target.events[eind, ...] + if np.logical_and(1 <= extremes, extremes <= 15).any(): + extreme = True + else: + extreme = False + + # set mask = 1 where data are OK + cloud_mask = target.cloudmask_en[tind, ...] < 1.0 + lc = target.SCL[tind, ...] + lc_mask = lc == 4 # lc != 4 #(lc <= 40) | (lc >= 90) + # lc_mask = np.expand_dims(lc_mask, axis=0) + # lc_mask = np.repeat(lc_mask, target_len, axis=0) + + mask = cloud_mask * lc_mask + + targ = np.where(targ * mask == 0, np.nan, targ) + pred = np.where(pred * mask == 0, np.nan, pred) + n_obs = np.count_nonzero(~np.isnan(targ), axis=0) # per pixel in space + + if np.sum(n_obs) / (128 * 128 * target_len) < 0.05: + # not enough data + return cubename, season, extreme, None, None, None, round(np.sum(n_obs) / (128 * 128 * target_len)*100, ndigits=3) + elif subset == "extreme" and not extreme: + # extreme subset should not have non extreme obs. + return cubename, season, extreme, None, None, None, None + elif subset == "non_extreme" and extreme: + # non extreme subset should not have extreme obs. + return cubename, season, extreme, None, None, None, None + #else: + + if method == "by_frame": + # stats are first computed over time and then averaged over space by minicube + sum_squared_error = np.nansum((targ - pred) ** 2, axis=0) + rmse = np.sqrt(sum_squared_error / (n_obs + 1e-8)) + + r2 = np.zeros((128, 128)) + # cov = np.zeros((128, 128)) + sum_squared_dev = np.zeros((128, 128)) + for i in range(128): + for j in range(128): + x = targ[:, i, j] + y = pred[:, i, j] + nas = np.logical_or(np.isnan(x), np.isnan(y)) + if ( + len(x[~nas]) > 1 + and ~np.all(x[~nas] == x[~nas][0]) + and ~np.all(y[~nas] == y[~nas][0]) + ): + x = x[~nas] + y = y[~nas] + + mean_targ = np.mean(x) + mean_pred = np.mean(y) + + dev_targ = x - mean_targ + dev_pred = y - mean_pred + + dev_targ_squared = dev_targ**2 + dev_pred_squared = dev_pred**2 + + r = np.sum(dev_targ * dev_pred) / ( + np.sqrt(np.sum(dev_pred_squared)) + * np.sqrt(np.sum(dev_targ_squared)) + + 1e-8 + ) + r2[i, j] = r**2 + + # cov[i, j] = np.sum(dev_targ * dev_targ) / ( + # np.sqrt(np.sum(dev_targ_squared)) + # * np.sqrt(np.sum(dev_targ_squared)) + # + 1e-8 + # ) + sum_squared_dev[i,j] = np.sum(dev_targ_squared) + else: + sum_squared_dev[i, j] = np.nan + r2[i, j] = np.nan + + # if len(x[~nas]) > 1 and ~np.all(x[~nas] == x[~nas][0]): + # r2[i, j] = (pearsonr(x[~nas], y[~nas]).statistic) ** 2 + # cov[i, j] = (pearsonr(x[~nas], x[~nas]).statistic) ** 2 + # else: + # if np.count_nonzero(~np.isnan(x)) < 2: + # cov[i, j] = np.nan + # r2[i, j] = np.nan + # else: + # cov[i, j] = 1 + # if x[~nas].all() == y[~nas].all(): + # r2[i, j] = 1 + # else: + # r2[i, j] = np.nan + + # nse = 1 - (sum_squared_error / (cov**2 + 1e-8)) + nse = 1 - (sum_squared_error / (sum_squared_dev + 1e-8)) + + if np.count_nonzero(~np.isnan(rmse)) == 0: + rmse = None + else: + rmse = np.nanmean(rmse) + if np.count_nonzero(~np.isnan(nse)) == 0: + nse = None + else: + nse = np.nanmean(nse) + if np.count_nonzero(~np.isnan(r2)) == 0: + r2 = None + else: + r2 = np.nanmean(r2) + return cubename, season, extreme, rmse, nse, r2, round(np.sum(n_obs) / (128 * 128 * target_len)*100, ndigits=3) + elif method == "overall": + sum_squared_error = np.nansum((targ - pred) ** 2) + rmse = np.sqrt(sum_squared_error / (np.sum(n_obs) + 1e-8)) + nas = np.logical_or(np.isnan(targ), np.isnan(pred)) + r2 = (pearsonr(targ[~nas], pred[~nas]).statistic) ** 2 + sum_squared_dev = np.nansum((targ - np.nanmean(targ)) ** 2) + nse = 1 - (sum_squared_error / (sum_squared_dev + 1e-8)) + return cubename, season, extreme, rmse, nse, r2, round(np.sum(n_obs) / (128 * 128 * target_len)*100, ndigits=3) + else: + raise NameError("Method " + method +" is not defined.") + +def get_season(date:np.datetime64) -> str: + # 1: MAM, 2: JJA, 3: SON, 4: DJF + seasons = ['MAM', 'JJA', 'SON', 'DJF',] + season = int(str(np.datetime64(date, 'M'))[5:]) // 3 -1 + return seasons[3 if season < 0 else season] + + +def get_name(path: Path) -> str: + """Helper function gets Cubename from a Path + + Args: + path (Path): One of Path/to/cubename.npz and Path/to/experiment_cubename.npz + + Returns: + [str]: cubename (has format mc_lon_lat) + """ + components = path.name.split("/") + cubename = components[-1][0:-16] + season = get_season(np.datetime64(components[-1][-15:-5])) + return cubename, season + + +def calculate_sample_statistics(pred_path, anomalie=False, method="by_frame"): + cubename, season = get_name(pred_path) + try : + pred = xr.open_dataset(pred_path, engine="zarr").load() + # get test paths + test_path = list(Path("/Net/Groups/BGI/tscratch/mweynants/dx-minicubes").glob("full/*/" + cubename + "*.zarr"))[0] + target = xr.open_dataset(test_path, engine="zarr").load() + + if anomalie: + pred_anomalie_path = list( + Path( + "/Net/Groups/BGI/scratch/crobin/PythonProjects/EarthNet/earthnet-models-pytorch/experiments/en23/convlstm_ae/convlstm_ae/config_16.10.2023_anomalie_NDVI/preds/iid" + ).glob("*/" + name) + )[0] + pred_anomalie = xr.open_dataset(pred_anomalie_path, engine="zarr").load() + return [ + calculate_variable_statistics(target, prediction, anomalie, "all", cubename, method, season,) + for prediction, anomalie in [(pred, False), (pred_anomalie, True)] + ] + else: + pred_anomalie = None + return [ + calculate_variable_statistics(target, prediction, anomalie, "all", cubename, method, season,) + for prediction, anomalie in [(pred, False)] + ] + except : + return [(cubename, season, None, None, None, None, None)] + +def calculate(args): + result = calculate_sample_statistics(*args) + return result + +def process_samples_in_parallel(paths, anomalie=False, method = "by_frame"): + "Function to process samples using parallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + tasks = [(path, anomalie, method) for path in paths] + results = list( + # tqdm(pool.imap(calculate_sample_statistics, paths,), total=len(paths)) + tqdm(pool.imap(calculate, tasks), total=len(tasks)) + ) + pool.close() + pool.join() + return results + + +if __name__ == "__main__": + parser = ArgumentParser() + parser.add_argument( + "bpath", + type=str, + metavar="path/to/experiment", + help="experiments results", + ) + # bpath = Path( + # "/Net/Groups/BGI/scratch/mweynants/DeepExtremes/earthnet-models-pytorch/experiments/de23/convlstm_ae/convlstm_ae/config_lr_1e-4_mlst50" + # ) + parser.add_argument( + "output", + type=str, + metavar="path/to/output", + help="path to save output json file", + ) + + parser.add_argument( + "method", + type=str, + metavar="by_frame", + help="metrics computation method. 'by_frame' or 'overall'" + ) + + args = parser.parse_args() + + bpath = args.bpath + components = bpath.split("/") + output_json = "results_" + components[-4] + "_" + components[-2] + "_" + components[-1] + "_" + args.method + ".json" + print(Path(args.output, output_json)) + + basepath = Path( + bpath, "preds/iid/" + ) + + # paths to test predictions + paths = list(basepath.glob("*.zarr")) + + print("len of the dataset: ", len(paths)) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(paths, method = args.method) + + # Transpose the results to get statistics for each variable + variable_statistics = list(map(list, zip(*sample_statistics))) + + # save the statistics for each variable + data = {} + for var_idx, var_stats in enumerate(variable_statistics): + # var_name = ["absolute_ndvi", "anomalie_ndvi"][var_idx] + cubename, season, extreme, rmse, nse, r2, valid_pixels = zip(*var_stats) + rmse = np.float64(rmse).tolist() + nse = np.float64(nse).tolist() + r2 = np.float64(r2).tolist() + # data[str(var_name)] = { + data = { + "cubename": [i for i in cubename], + "season": [i for i in season], + "extreme": [i for i in extreme], + "rmse": [i for i in rmse], # [i for i in rmse if i is not (None or np.nan)], + "nse": [i for i in nse], # [i for i in nse if i is not (None or np.nan)], + "r2": [i for i in r2], # [i for i in r2 if i is not (None or np.nan)], + "valid_pixels": [i for i in valid_pixels] + } + + with open(Path(args.output, output_json), "w") as fp: + json.dump(data, fp) + + # print(data) + print("output written to: " + str(Path(args.output, output_json))) + diff --git a/scripts/postprocessing/de23_metric_computation2.py b/scripts/postprocessing/de23_metric_computation2.py new file mode 100644 index 0000000..97e4f5e --- /dev/null +++ b/scripts/postprocessing/de23_metric_computation2.py @@ -0,0 +1,321 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import json +from tqdm import tqdm +import multiprocessing as mp +import os +from scipy.stats import pearsonr +from argparse import ArgumentParser +import sys + +era5 = [ + "t2m_mean", + "pev_mean", + "slhf_mean", + "ssr_mean", + "sp_mean", + "sshf_mean", + "e_mean", + "tp_mean", + "t2m_min", + "pev_min", + "slhf_min", + "ssr_min", + "sp_min", + "sshf_min", + "e_min", + "tp_min", + "t2m_max", + "pev_max", + "slhf_max", + "ssr_max", + "sp_max", + "sshf_max", + "e_max", + "tp_max", +] + + +def calculate_variable_statistics( + target, + pred, + cubename="", + method="by_frame", + season="DJF", +): + """Function to calculate statistics for each variable in a sample""" + t0 = pred.time.values[0] + target_len = len(pred.time) # - 1 fixing mistake len prediction iid test set... + # time start index + it = np.where(target.time == t0)[0][0] + # first target_len steps, every 5 days + tind = range(it, it + target_len, 1) + + targ = (target.B8A - target.B04) / (target.B8A + target.B04 + 1e-8)[tind, ...] + pred = pred.ndvi_pred # [:-1, ...] + + # set mask = 1 where data are OK + cloud_mask = target.cloudmask_en[tind, ...] == 0 + lc = target.SCL[tind, ...] + lc_mask = lc == 4 + mask = cloud_mask * lc_mask + + targ = np.where(targ * mask == 0, np.nan, targ) + pred = np.where(pred * mask == 0, np.nan, pred) + n_obs = np.count_nonzero(~np.isnan(targ), axis=0) + if np.sum(n_obs) / (128 * 128 * target_len) < 0.05: + # not enough data + return ( + cubename, + season, + {}, + round(np.sum(n_obs) / (128 * 128 * target_len) * 100, ndigits=3), + ) + # Meteorological variables + era5_stats = {} + for variable in era5: + if variable[-3:] == "min": + era5_stats[variable] = np.float64(np.nanmin(target[variable][tind])) + elif variable[-3:] == "max": + era5_stats[variable] = np.float64(np.nanmax(target[variable][tind])) + else: + era5_stats[variable] = np.float64(np.nanmean(target[variable][tind])) + + # NDVI + # mean_ndvi = np.nansum(targ) / n_obs + # std_ndvi = np.sqrt(np.nansum(targ - mean_ndvi)) + if method == "by_frame": + # stats are first computed over time and then averaged over space by minicube + sum_squared_error = np.nansum((targ - pred) ** 2, axis=0) + rmse = np.sqrt(sum_squared_error / (n_obs + 1e-8)) + + r2 = np.zeros((128, 128)) + mean_targ = np.zeros((128, 128)) + sum_squared_dev = np.zeros((128, 128)) + for i in range(128): + for j in range(128): + x = targ[:, i, j] + y = pred[:, i, j] + nas = np.logical_or(np.isnan(x), np.isnan(y)) + if ( + len(x[~nas]) > 1 + and ~np.all(x[~nas] == x[~nas][0]) + and ~np.all(y[~nas] == y[~nas][0]) + ): + x = x[~nas] + y = y[~nas] + + mean = np.mean(x) + dev_targ = x - mean + + dev_targ_squared = dev_targ**2 + + mean_targ[i, j] = mean + r2[i, j] = (pearsonr(x, y).statistic) ** 2 + sum_squared_dev[i, j] = np.sum(dev_targ_squared) + else: + sum_squared_dev[i, j] = np.nan + r2[i, j] = np.nan + + nse = 1 - (sum_squared_error / (sum_squared_dev + 1e-8)) + nnse = 1 / (2 - nse) + rmse = np.nanmean(rmse) if np.count_nonzero(~np.isnan(rmse)) != 0 else None + nnse = np.nanmean(nnse) if np.count_nonzero(~np.isnan(nnse)) != 0 else None + nse = np.nanmedian(nse) if np.count_nonzero(~np.isnan(nse)) != 0 else None + r2 = np.nanmean(r2) if np.count_nonzero(~np.isnan(r2)) != 0 else None + mean_targ = ( + np.nanmean(mean_targ) + if np.count_nonzero(~np.isnan(mean_targ)) != 0 + else None + ) + sum_squared_dev = ( + np.nanmean(sum_squared_dev) + if np.count_nonzero(~np.isnan(sum_squared_dev)) != 0 + else None + ) + + result = { + "rmse": np.float64(rmse), + "nse": np.float64(nse), + "nnse": np.float64(nnse), + "r2": np.float64(r2), + "mean_targ": np.float64(mean_targ), + "sigma_squared_targ": np.float64(sum_squared_dev), + } + result.update(era5_stats) + + return ( + cubename, + season, + result, + round(np.sum(n_obs) / (128 * 128 * target_len) * 100, ndigits=3), + ) + elif method == "overall": + sum_squared_error = np.nansum((targ - pred) ** 2) + rmse = np.sqrt(sum_squared_error / (np.sum(n_obs) + 1e-8)) + nas = np.logical_or(np.isnan(targ), np.isnan(pred)) + + r2 = (pearsonr(targ[~nas], pred[~nas]).statistic) ** 2 + sum_squared_dev = np.nansum((targ - np.nanmean(targ)) ** 2) + nse = 1 - (sum_squared_error / (sum_squared_dev + 1e-8)) + + result = { + "rmse": np.float64(rmse), + "nse": np.float64(nse), + "r2": np.float64(r2), + } + + return ( + cubename, + season, + result, + round(np.sum(n_obs) / (128 * 128 * target_len) * 100, ndigits=3), + ) + else: + raise NameError("Method " + method + " is not defined.") + + +def get_season(date: np.datetime64) -> str: + # 1: MAM, 2: JJA, 3: SON, 4: DJF + seasons = [ + "MAM", + "JJA", + "SON", + "DJF", + ] + season = int(str(np.datetime64(date, "M"))[5:]) // 3 - 1 + return seasons[3 if season < 0 else season] + + +def get_name(path: Path) -> str: + """Helper function gets Cubename from a Path + + Args: + path (Path): One of Path/to/cubename.npz and Path/to/experiment_cubename.npz + + Returns: + [str]: cubename (has format mc_lon_lat) + """ + components = path.name.split("/") + cubename_target = components[-1][0:-16] + cubename_pred = components[-1] + season = get_season(np.datetime64(components[-1][-15:-5])) + return cubename_target, cubename_pred, season + + +def calculate_sample_statistics(pred_path, method="by_frame"): + cubename_target, cubename_pred, season = get_name(pred_path) + # try: + pred = xr.open_dataset(pred_path, engine="zarr").load() + # get test paths + test_path = list( + Path("/Net/Groups/BGI/tscratch/mweynants/dx-minicubes").glob( + "full/*/" + cubename_target + "*.zarr" + ) + )[0] + target = xr.open_dataset(test_path, engine="zarr").load() + return calculate_variable_statistics(target, pred, cubename_pred, method, season) + + +def calculate(args): + result = calculate_sample_statistics(*args) + return result + + +def process_samples_in_parallel(paths, method="by_frame"): + "Function to process samples using parallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + tasks = [(path, method) for path in paths] + results = list( + # tqdm(pool.imap(calculate_sample_statistics, paths,), total=len(paths)) + tqdm(pool.imap(calculate, tasks), total=len(tasks)) + ) + pool.close() + pool.join() + return results + + +if __name__ == "__main__": + parser = ArgumentParser() + parser.add_argument( + "bpath", + type=str, + metavar="path/to/experiment", + help="experiments results", + ) + # bpath = Path( + # "/Net/Groups/BGI/scratch/mweynants/DeepExtremes/earthnet-models-pytorch/experiments/de23/convlstm_ae/convlstm_ae/config_lr_1e-4_mlst50" + # ) + parser.add_argument( + "output", + type=str, + metavar="path/to/output", + help="path to save output json file", + ) + + parser.add_argument( + "method", + type=str, + metavar="by_frame", + help="metrics computation method. 'by_frame' or 'overall'", + ) + + parser.add_argument( + "test_set", + type=str, + metavar="iid", + ) + + args = parser.parse_args() + + path_list = args.bpath.split("/")[:-2] + bpath = "/".join(path_list) + + components = bpath.split("/") + output_json = ( + "results_" + + components[-4] + + "_" + + components[-2] + + "_" + + components[-1] + + "_" + + args.test_set + + "_" + + args.method + + ".json" + ) + print(Path(args.output, output_json)) + + basepath = Path(bpath, "preds/" + args.test_set + "/") + + # paths to test predictions + paths = list(basepath.glob("*.zarr")) + + print("len of the dataset: ", len(paths)) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(paths, method=args.method) + # Transpose the results to get statistics for each variable + # variable_statistics = list(map(list, zip(*sample_statistics))) + # save the statistics for each variable + data = [] + for sample in sample_statistics: + ( + cubename, + season, + results, + valid_pixels, + ) = sample + sample_dict = {"name": cubename, "season": season, "valid_pixels": valid_pixels} + sample_dict.update(results) + data.append(sample_dict) + + with open(Path(args.output, output_json), "w") as fp: + json.dump(data, fp) + + # print(data) + print("output written to: " + str(Path(args.output, output_json))) diff --git a/scripts/postprocessing/en23_metric_computation.py b/scripts/postprocessing/en23_metric_computation.py new file mode 100644 index 0000000..620d197 --- /dev/null +++ b/scripts/postprocessing/en23_metric_computation.py @@ -0,0 +1,194 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import json +from tqdm import tqdm +import multiprocessing as mp +import os +from scipy.stats import pearsonr + + +def calculate_variable_statistics(minicube, pred, anomalie=False, subset="all"): + """Function to calculate statistics for each variable in a sample""" + time = [minicube.time.values[i] for i in range(4, 450, 5)] + context_length = 60 + target_length = 10 + target = (minicube.s2_B8A - minicube.s2_B04) / ( + minicube.s2_B8A + minicube.s2_B04 + 1e-8 + ) + if anomalie: + target = target - minicube.msc + targ = target.sel(time=time)[context_length : context_length + target_length, ...] + pred = pred.ndvi_pred + + extremes = minicube.extremes[ + context_length * 5 : (context_length + target_length) * 5, ... + ] + if np.logical_and(1 <= extremes, extremes <= 15).any(): + extreme = True + else: + extreme = False + + cloud_mask = ( + minicube.s2_mask.sel(time=time)[ + context_length : context_length + target_length, ... + ] + < 1.0 + ) + lc = minicube.esawc_lc + + lc_mask = (lc <= 40) | (lc >= 90) + lc_mask = np.expand_dims(lc_mask, axis=0) + lc_mask = np.repeat(lc_mask, 10, axis=0) + + mask = cloud_mask * lc_mask + + targ = np.where(targ * mask == 0, np.nan, targ) + pred = np.where(pred * mask == 0, np.nan, pred) + n_obs = np.count_nonzero(~np.isnan(targ), axis=0) # per pixel in space + if ( + (np.sum(n_obs) / (pred.shape[0] * pred.shape[1] * pred.shape[2]) < 0.1) + or (subset == "extreme" and not extreme) + or (subset == "non_extreme" and extreme) + ): + return None, None, None, None + + else: + lc_mask = (lc <= 40) | (lc >= 90) + landcover = np.where(lc * lc_mask == 0, np.nan, lc) + # Use numpy's unique function to get unique elements and their counts + unique_elements, counts = np.unique( + landcover[~np.isnan(landcover)].flatten(), return_counts=True + ) + + # Find the index of the maximum count + + max_count_index = np.argmax(counts) + + # The most common number + lc_maj = unique_elements[max_count_index] + + sum_squared_error = np.nansum((targ - pred) ** 2, axis=0) + + mse = sum_squared_error / (n_obs + 1e-8) + + r2 = np.zeros((128, 128)) + sum_squared_dev = np.zeros((128, 128)) + for i in range(128): + for j in range(128): + x = targ[:, i, j] + y = pred[:, i, j] + + nas = np.logical_or(np.isnan(x), np.isnan(y)) + if ( + len(x[~nas]) > 1 + and ~np.all(x[~nas] == x[~nas][0]) + and ~np.all(y[~nas] == y[~nas][0]) + ): + x = x[~nas] + y = y[~nas] + + mean_targ = np.mean(x) + mean_pred = np.mean(y) + + dev_targ = x - mean_targ + dev_pred = y - mean_pred + + dev_targ_squared = dev_targ**2 + dev_pred_squared = dev_pred**2 + + r = np.sum(dev_targ * dev_pred) / ( + np.sqrt(np.sum(dev_pred_squared)) + * np.sqrt(np.sum(dev_targ_squared)) + + 1e-8 + ) + r2[i, j] = r**2 + sum_squared_dev[i, j] = np.sum(dev_targ_squared) + + else: + r2[i, j] = np.nan + sum_squared_dev[i, j] = np.nan + + nse = 1 - (sum_squared_error / (sum_squared_dev + 1e-8)) + mse = np.nanmean(mse) if np.count_nonzero(~np.isnan(mse)) != 0 else None + nse = np.nanmedian(nse) if np.count_nonzero(~np.isnan(nse)) != 0 else None + r2 = np.nanmean(r2) if np.count_nonzero(~np.isnan(r2)) != 0 else None + return mse, nse, r2, lc_maj + + +def calculate_sample_statistics(pred_path): + name = str(pred_path)[-12:] + test_path = list( + Path("/scratch/crobin/earthnet2023_preprocessing/test/").glob("*/" + name) + )[0] + pred_anomalie_path = list( + Path( + "/Net/Groups/BGI/scratch/crobin/PythonProjects/EarthNet/earthnet-models-pytorch/experiments/en23/convlstm_ae/convlstm_ae/config_16.10.2023_anomalie_NDVI/preds/iid" + ).glob("*/" + name) + )[0] + target = xr.open_dataset(test_path, engine="netcdf4").load() + pred = xr.open_dataset(pred_path, engine="netcdf4").load() + pred_anomalie = xr.open_dataset(pred_anomalie_path, engine="netcdf4").load() + + return [ + calculate_variable_statistics(target, prediction, anomalie, "non_extreme") + for prediction, anomalie in [(pred, False), (pred_anomalie, True)] + ] + + +def process_samples_in_parallel(paths): + "Function to process samples using parallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + results = list( + tqdm(pool.imap(calculate_sample_statistics, paths), total=len(paths)) + ) + pool.close() + pool.join() + return results + + +if __name__ == "__main__": + basepath = Path( + "/Net/Groups/BGI/scratch/crobin/PythonProjects/EarthNet/earthnet-models-pytorch/experiments/en23/convlstm_ae/convlstm_ae/config_16.10.2023_absolute_NDVI/preds/iid/" + ) + + paths = list(basepath.glob("*/*.nc")) + print("len of the dataset: ", len(paths)) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(paths) + + # Transpose the results to get statistics for each variable + variable_statistics = list(map(list, zip(*sample_statistics))) + + # save the statistics for each variable + data, total = {}, {} + + for var_idx, var_stats in enumerate(variable_statistics): + var_name = ["absolute_ndvi", "anomalie_ndvi"][var_idx] + + mse, nse, r2, lc_maj = zip(*var_stats) + mse = np.float64(mse).tolist() + nse = np.float64(nse).tolist() + r2 = np.float64(r2).tolist() + lc_maj = np.float64(lc_maj) + + data[str(var_name)] = { + "rmse": [np.sqrt(i) for i in mse if i is not (None or np.nan)], + "nse": [i for i in nse if i is not (None or np.nan)], + "r2": [i for i in r2 if i is not (None or np.nan)], + "lc_maj": [i for i in lc_maj if i is not (None or np.nan)], + } + total[str(var_name)] = { + "rmse": np.sqrt(np.nanmean(mse)), + "nse": np.nanmedian(nse), + "r2": np.nanmean(r2), + } + name_file = "results_test.json" + with open("results_test_non_extreme_final.json", "w") as fp: + json.dump(data, fp) + + with open("total_extreme_" + name_file, "w") as fp: + json.dump(total, fp) diff --git a/scripts/postprocessing/era5_trainset.py b/scripts/postprocessing/era5_trainset.py new file mode 100644 index 0000000..0ad895a --- /dev/null +++ b/scripts/postprocessing/era5_trainset.py @@ -0,0 +1,271 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import json +from tqdm import tqdm +import multiprocessing as mp +import os +from scipy.stats import pearsonr +from argparse import ArgumentParser +import sys +import pandas as pd + +import zarr +import pandas as pd +from pathlib import Path +import datetime +import xarray as xr +import numpy as np +from tqdm import tqdm +import random +import multiprocessing as mp +from scipy.interpolate import interp1d +import os +import sys +import pandas as pd +import geopandas as gpd +from shapely.geometry import Point + +era5 = [ + "t2m_mean", + "pev_mean", + "slhf_mean", + "ssr_mean", + "sp_mean", + "sshf_mean", + "e_mean", + "tp_mean", + "t2m_min", + "pev_min", + "slhf_min", + "ssr_min", + "sp_min", + "sshf_min", + "e_min", + "tp_min", + "t2m_max", + "pev_max", + "slhf_max", + "ssr_max", + "sp_max", + "sshf_max", + "e_max", + "tp_max", +] + + +def calculate_variable_statistics( + target, + cubename="", + method="by_frame", +): + """Function to calculate statistics for each variable in a sample""" + # set mask = 1 where data are OK + cloud_mask = target.cloudmask_en == 0 + lc = target.SCL + lc_mask = lc == 4 + mask = cloud_mask * lc_mask + targ = (target.B8A - target.B04) / (target.B8A + target.B04 + 1e-8) + + targ = np.where(targ * mask == 0, np.nan, targ) + n_obs = np.count_nonzero(~np.isnan(targ), axis=0) + if np.sum(n_obs) / (128 * 128 * 17) < 0.05: + # not enough data + return ( + cubename, + {}, + ) + mean_targ = np.zeros((128, 128)) + sum_squared_dev = np.zeros((128, 128)) + for i in range(128): + for j in range(128): + x = targ[:, i, j] + + nas = np.isnan(x) + if len(x[~nas]) > 1 and ~np.all(x[~nas] == x[~nas][0]): + x = x[~nas] + mean = np.mean(x) + dev_targ = x - mean + dev_targ_squared = dev_targ**2 + mean_targ[i, j] = mean + sum_squared_dev[i, j] = np.sum(dev_targ_squared) + else: + sum_squared_dev[i, j] = np.nan + mean_targ = ( + np.nanmean(mean_targ) if np.count_nonzero(~np.isnan(mean_targ)) != 0 else None + ) + sum_squared_dev = ( + np.nanmean(sum_squared_dev) + if np.count_nonzero(~np.isnan(sum_squared_dev)) != 0 + else None + ) + + # Meteorological variables + era5_stats = {} + for variable in era5: + if variable[-3:] == "min": + era5_stats[variable] = np.float64(np.nanmin(target[variable])) + elif variable[-3:] == "max": + era5_stats[variable] = np.float64(np.nanmax(target[variable])) + else: + era5_stats[variable] = np.float64(np.nanmean(target[variable])) + + # NDVI + if method == "by_frame": + # stats are first computed over time and then averaged over space by minicube + result = { + "mean_targ": np.float64(mean_targ), + "sigma_squared_targ": np.float64(sum_squared_dev), + } + result.update(era5_stats) + return ( + cubename, + result, + ) + + +def calculate_sample_statistics(data, method="by_frame"): + cubename_target = data["path"] + end_date = data["end_date"] + start_date = end_date - datetime.timedelta(days=17 * 5) + + # try: + # get test paths + test_path = Path( + "/Net/Groups/BGI/tscratch/mweynants/dx-minicubes", cubename_target[1:] + ) + target = ( + xr.open_dataset(test_path, engine="zarr") + .sel(time=slice(start_date, end_date)) + .load() + ) + return calculate_variable_statistics(target, cubename_target, method) + + +def calculate(args): + result = calculate_sample_statistics(*args) + return result + + +def process_samples_in_parallel(paths, method="by_frame"): + "Function to process samples using parallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + tasks = [(paths.iloc[i], method) for i in range(paths.shape[0])] + results = list( + # tqdm(pool.imap(calculate_sample_statistics, paths,), total=len(paths)) + tqdm(pool.imap(calculate, tasks), total=len(tasks)) + ) + pool.close() + pool.join() + return results + + +def dataset_split_continent(test_fold, val_fold): + # Read your CSV file into a pandas DataFrame + # Read your CSV file into a pandas DataFrame + fold_path = "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet_biome.csv" + df = pd.read_csv(fold_path, delimiter=",")[ + ["path", "group", "check", "start_date", "lat", "lon", "biome"] + ] + + # 3 training minicubes between 2017 - 2020 (randomly starting in the first year), followed by 3 test minicubes in 2021 + df["start_date"] = pd.to_datetime(df["start_date"], format="%Y-%m-%dT%H:%M:%S.%f") + df["start_date2"] = df["start_date"] + datetime.timedelta(days=450) + df["start_date3"] = df["start_date2"] + datetime.timedelta(days=450) + df["start_test1"] = df["start_date3"] + datetime.timedelta(days=450) + df["start_test2"] = df["start_test1"] + datetime.timedelta(days=90) + df["start_test3"] = df["start_test2"] + datetime.timedelta(days=90) + df = df.melt( + ["path", "group", "check", "lat", "lon", "biome"], value_name="start_date" + ) + df["end_date"] = df["start_date"] + datetime.timedelta(days=449) + + # Check if the point is in Africa + def is_in_africa(row, africa): + # Create a Shapely Point from latitude and longitude + point = Point(row["lon"], row["lat"]) + + # Check if the point is in Africa + return africa.contains(point).any() + + # Load a world map dataset (included in geopandas) + world = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres")) + africa = world[world["continent"] == "Africa"] + # Apply the is_in_africa function to each row + df["in_africa"] = df.apply(lambda row: is_in_africa(row, africa), axis=1) + + cols = ["path", "start_date", "end_date", "biome", "in_africa"] + # temporal test set 2021 + temporal_test_subset = df.loc[ + (df["variable"].str.startswith("start_test")), + cols, + ] + + # folds 2017 - 2020 + df = df.loc[df["variable"].str.startswith("start_date")].drop("variable", 1) + + # training set + train_subset = df.loc[ + (df["group"] != test_fold) + & (df["group"] != val_fold) + & (df["check"] == 0) + & (df["in_africa"] == False), + cols, + ] + + # validation set + val_subset = df.loc[ + (df["group"] == val_fold) & (df["check"] == 0) & (df["in_africa"] == False), + cols, + ] + + # iid test set + spatial_test_subset = df.loc[ + (df["group"] == test_fold) & (df["check"] == 0) & (df["in_africa"] == False), + cols, + ] + + africa_test_subset = df.loc[ + (df["check"] == 0) & (df["in_africa"] == True), + cols, + ] + + return ( + train_subset, + val_subset, + spatial_test_subset, + temporal_test_subset, + africa_test_subset, + ) + + +if __name__ == "__main__": + + output_json = "/Net/Groups/BGI/scratch/crobin/PythonProjects/EarthNet/Data_analysis/ML4RS/results_trainset_ndvi.json" + + train_subset, _, _, _, _ = dataset_split_continent(10, 9) + + print("len of the dataset: ", train_subset.shape[0]) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(train_subset) + # Transpose the results to get statistics for each variable + # variable_statistics = list(map(list, zip(*sample_statistics))) + # save the statistics for each variable + data = [] + for sample in sample_statistics: + ( + cubename, + results, + ) = sample + sample_dict = {"path": cubename} + sample_dict.update(results) + data.append(sample_dict) + + with open(output_json, "w") as fp: + json.dump(data, fp) + + # print(data) + print("output written to: " + output_json) diff --git a/scripts/preprocessing/add_biome.py b/scripts/preprocessing/add_biome.py new file mode 100644 index 0000000..cbe8841 --- /dev/null +++ b/scripts/preprocessing/add_biome.py @@ -0,0 +1,45 @@ +import geopandas as gpd +from shapely.geometry import Point +import pandas as pd +import time + +# Read your CSV file +df = pd.read_csv( + "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet_biome.csv", delimiter="," +) + +# Function to get biome from a row +# Load the WWF biome dataset +biome_data_path = "/Net/Groups/BGI/scratch/crobin/PythonProjects/EarthNet/terr-ecoregions-TNC/tnc_terr_ecoregions.shp" +biome_data = gpd.read_file(biome_data_path) + + +def get_biome(row): + point = Point(row["lon"], row["lat"]) + + for _, row in biome_data.iterrows(): + if point.within(row["geometry"]): + return row["WWF_MHTNAM"] + + return "Unknown" + + +# Load a world map dataset (included in geopandas) +world = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres")) +africa = world[world["continent"] == "Africa"] + + +def is_in_africa(row, africa): + # Create a Shapely Point from latitude and longitude + point = Point(row["lon"], row["lat"]) + + # Check if the point is in Africa + return africa.contains(point).any() + + +# Apply the is_in_africa function to each row +df["in_africa"] = df.apply(lambda row: is_in_africa(row, africa), axis=1) + +# Apply the function to create a new 'biome' column +# df["biome"] = df.apply(get_biome, axis=1) +df.to_csv("/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet_biome.csv") diff --git a/scripts/preprocessing/clean_dataset.py b/scripts/preprocessing/clean_dataset.py new file mode 100644 index 0000000..9fff754 --- /dev/null +++ b/scripts/preprocessing/clean_dataset.py @@ -0,0 +1,68 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import json +import datetime +from tqdm import tqdm +import multiprocessing as mp +import os +from scipy.stats import pearsonr +from argparse import ArgumentParser +import sys +import pandas as pd + + + +def is_valide( + raw +): + """Function to calculate statistics for each variable in a sample""" + filepath = raw['filepath'] + t0 = pred.time.values[0] + target_len = 13 + # time start index + it = np.where(target.time == t0)[0][0] + # first target_len steps, every 5 days + tind = range(it, it + target_len, 1) + + targ = (target.B8A - target.B04) / (target.B8A + target.B04 + 1e-8)[tind, ...] + + # set mask = 1 where data are OK + cloud_mask = target.cloudmask_en[tind, ...] == 0 + lc = target.SCL[tind, ...] + lc_mask = (lc == 4) | (lc == 4) + mask = cloud_mask * lc_mask + + targ = np.where(targ * mask == 0, np.nan, targ) + n_obs = np.count_nonzero(~np.isnan(targ), axis=0) + if np.sum(n_obs) / (128 * 128 * target_len) > 0.05: + return True + else: + return False + + +if __name__ == "__main__": + # Read your CSV file + df = pd.read_csv( + "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet_biome.csv", + delimiter=",", + ) + df["start_date"] = pd.to_datetime( + df["start_date"], format="%Y-%m-%dT%H:%M:%S.%f" + ) + df["start_date2"] = df["start_date"] + datetime.timedelta(days=450) + df["start_date3"] = df["start_date2"] + datetime.timedelta(days=450) + df["start_test1"] = df["start_date3"] + datetime.timedelta(days=450) + df["start_test2"] = df["start_test1"] + datetime.timedelta(days=90) + df["start_test3"] = df["start_test2"] + datetime.timedelta(days=90) + df = df.melt(["path", "group", "check", "lat", "lon"], value_name="start_date") + df["end_date"] = df["start_date"] + datetime.timedelta(days=449) + + paths = list( + Path("/Net/Groups/BGI/tscratch/mweynants/dx-minicubes").glob("full/*/*.zarr") + )[0] + print("len paths: ", len(paths)) + output = "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet_biome.csv") + + df["is_valid"] = df.apply(lambda row: is_valid(row), axis=1) + df.to_csv(output) diff --git a/scripts/preprocessing/de23_add_interpolation.py b/scripts/preprocessing/de23_add_interpolation.py new file mode 100644 index 0000000..6b3234f --- /dev/null +++ b/scripts/preprocessing/de23_add_interpolation.py @@ -0,0 +1,164 @@ +import zarr +import pandas as pd +from pathlib import Path +import datetime +import xarray as xr +import numpy as np +from tqdm import tqdm +import random +import multiprocessing as mp +from scipy.interpolate import interp1d +import os +import sys + +spatio_temporal_variables = ["B02", "B03", "B04", "B05", "B06", "B07", "B8A"] + + +def get_dataset(): + test_fold = 10 + val_fold = 9 + + # load csv + df = pd.read_csv( + "/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv", delimiter="," + )[["path", "group", "check", "start_date"]] + + # 3 training minicubes between 2017 - 2020 (randomly starting in the first year), followed by 3 test minicubes in 2021 + df["start_date"] = pd.to_datetime(df["start_date"], format="%Y-%m-%dT%H:%M:%S.%f") + df["start_date2"] = df["start_date"] + datetime.timedelta(days=450) + df["start_date3"] = df["start_date2"] + datetime.timedelta(days=450) + df["start_test1"] = df["start_date3"] + datetime.timedelta(days=450) + df["start_test2"] = df["start_test1"] + datetime.timedelta(days=90) + df["start_test3"] = df["start_test2"] + datetime.timedelta(days=90) + df = df.melt( + ["path", "group", "check"], value_name="start_date", var_name="variable" + ) + # use only the context period + df["end_date"] = df["start_date"] + datetime.timedelta(days=364) + + # temporal test set 2021 + temporal_test_subset = df.loc[ + (df["variable"].str.startswith("start_test")), + ["path", "start_date", "end_date"], + ] + + # folds 2017 - 2020 + df = df.loc[df["variable"].str.startswith("start_date")].drop(columns="variable") + + # training set + train_subset = df.loc[ + (df["group"] != test_fold) & (df["group"] != val_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ] + + # validation set + val_subset = df.loc[ + (df["group"] == val_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ] + + # iid test set + spatial_test_subset = df.loc[ + (df["group"] == test_fold) & (df["check"] == 0), + ["path", "start_date", "end_date"], + ] + + return train_subset, val_subset, spatial_test_subset, temporal_test_subset + + +def interpolate_temporal(data): + # Get the indices along the time_steps dimension + time_indices = np.arange(len(data)) + # Create a mask for non-missing values + mask = ~np.isnan(data) + + if mask.any(): + # If the first frame has missing values, fill them with the mean of the frame + if ~mask[0]: + data[0] = np.nanmean(data) + mask[0] = True + # If the last frame has missing values, fill them with the mean of the frame + if ~mask[-1]: + data[-1] = np.nanmean(data) + mask[-1] = True + + # Use linear interpolation to fill missing values + interp_func = interp1d(time_indices[mask], data[mask], kind="linear") + data[~mask] = interp_func(time_indices[~mask]) + return data + + +def process_sample(metadata): + filepath, start_date, end_date = metadata + output_filepath = os.path.join(dst_path, str(filepath)[len(str(basepath)) + 1 :]) + if os.path.exists(output_filepath): + filepath = output_filepath + + minicube_original = xr.open_dataset(filepath, engine="zarr") + time = slice(start_date, end_date) + event_time = slice( + start_date, min(end_date, pd.Timestamp(datetime.date(2021, 12, 31))) + ) + minicube = minicube_original.sel( + time=time, + event_time=event_time, + ) + mask_cloud = minicube.cloudmask_en.values == 0 + mask_is_avail = minicube.SCL.values != 0 + + for variable in spatio_temporal_variables: + # Get the variable data from the minicube + var_data = minicube[variable].values + mask = mask_cloud * mask_is_avail + # Apply temporal interpolation to fill missing values + var_data = np.where(var_data * mask == 0, np.nan, var_data) + # Apply interpolation to all i, j positions simultaneously + if mask.any(): + interpolated_results = np.apply_along_axis( + interpolate_temporal, 0, var_data + ) + # Combine the interpolated data back into the minicube + minicube_original[variable].loc[ + dict( + time=time, + ) + ] = interpolated_results + + # Saving + minicube_original.chunk().to_zarr(output_filepath, mode="w", consolidated=True) + + +if __name__ == "__main__": + basepath = Path("/Net/Groups/BGI/tscratch/mweynants/dx-minicubes/") + dst_path = "/Net/Groups/BGI/tscratch/crobin/dx-minicubes_interpolated/" + + if not os.path.exists(dst_path): + os.makedirs(dst_path) + + folders = list(basepath.glob("full/*")) + + for folder in folders: + path = os.path.join(dst_path, str(folder)[len(str(basepath)) + 1 :]) + if not os.path.exists(path): + os.makedirs(path) + + ( + train_subset, + val_subset, + spatial_test_subset, + temporal_test_subset, + ) = get_dataset() + + metadata = [] + for metadata_files in [train_subset, val_subset, spatial_test_subset]: + metadata += sorted( + ( + Path(basepath, metadata_files["path"][idx][1:]), + metadata_files["start_date"][idx], + metadata_files["end_date"][idx], + ) + for idx in metadata_files.index + ) + + with mp.Pool(1) as pool: + r = list(tqdm(pool.imap(process_sample, metadata), total=len(metadata))) diff --git a/scripts/preprocessing/extremes.py b/scripts/preprocessing/en23_add_extremes.py old mode 100644 new mode 100755 similarity index 100% rename from scripts/preprocessing/extremes.py rename to scripts/preprocessing/en23_add_extremes.py diff --git a/scripts/preprocessing/interpolation_msc.py b/scripts/preprocessing/en23_add_interpolation_msc.py old mode 100644 new mode 100755 similarity index 100% rename from scripts/preprocessing/interpolation_msc.py rename to scripts/preprocessing/en23_add_interpolation_msc.py diff --git a/scripts/preprocessing/interpolation.py b/scripts/preprocessing/en23_interpolation.py old mode 100644 new mode 100755 similarity index 100% rename from scripts/preprocessing/interpolation.py rename to scripts/preprocessing/en23_interpolation.py diff --git a/scripts/preprocessing/nans_de23.py b/scripts/preprocessing/nans_de23.py new file mode 100755 index 0000000..912d7c4 --- /dev/null +++ b/scripts/preprocessing/nans_de23.py @@ -0,0 +1,113 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import pandas as pd +import json +from tqdm import tqdm +import multiprocessing as mp +import os + + +temporal_variables = [ + "t2m_mean", + "pev_mean", + "slhf_mean", + "ssr_mean", + "sp_mean", + "sshf_mean", + "e_mean", + "tp_mean", + "t2m_min", + "pev_min", + "slhf_min", + "ssr_min", + "sp_min", + "sshf_min", + "e_min", + "tp_min", + "t2m_max", + "pev_max", + "slhf_max", + "ssr_max", + "sp_max", + "sshf_max", + "e_max", + "tp_max", +] + +spatiotemporal_variables = [ + "B02", "B03", "B04", "B05", "B06", "B07", "B8A","SCL", "cloudmask_en" +] + +spatial_variables = ["cop_dem"] + +def calculate_variable_statistics(data): + """Function to calculate statistics for each variable in a sample""" + + max_value = np.nanmax(data) + min_value = np.nanmin(data) + sum_var = np.nansum(data) + n_obs = np.count_nonzero(~np.isnan(data)) + return max_value, min_value, sum_var, n_obs + +def calculate_nan(data): + """Function to calculate statistics for each variable in a sample""" + n_nan = np.count_nonzero(np.isnan(data)) + no_data = 1 if n_nan == data.shape[0] else 0 + return n_nan, no_data + +def calculate_sample_statistics(file): + minicube = xr.open_dataset(file, engine="zarr").load() + return [calculate_nan(minicube[variable].values) for variable in temporal_variables] + +def process_samples_in_parallel(paths): + "Function to process samples using pdata = minicube[variable].valuesarallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + results = list(tqdm(pool.imap(calculate_sample_statistics, paths), total=len(paths))) + pool.close() + pool.join() + return results + +def get_filepaths(metadata_file): + # load csv + df = pd.read_csv(metadata_file, delimiter=",")[["path", "check"]] + return df.loc[ (df["check"] == 0) , ["path"]].values.tolist() + + +if __name__ == "__main__": + basepath = "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" + + filepaths = get_filepaths("/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv") + + paths = [Path(basepath + fp[0]) for fp in filepaths] + + print(paths[:10]) + print("len of the dataset: ", len(paths)) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(paths) + + # Transpose the results to get statistics for each variable + variable_statistics = list(map(list, zip(*sample_statistics))) + + # save the statistics for each variable + data = {} + # for var_idx, var_stats in enumerate(variable_statistics): + # var_name = (temporal_variables)[var_idx] + # max_vals, min_vals, sum_vals, n_obs_vals = zip(*var_stats) + # mean = np.sum(sum_vals) / np.sum(n_obs_vals) + # data[str(var_name)] = {"mean": np.float64(mean), "min": np.float64(np.nanmin(min_vals)), "max": np.float64(np.nanmax(max_vals))} + + for var_idx, var_stats in enumerate(variable_statistics): + var_name = (spatiotemporal_variables + temporal_variables + spatial_variables)[var_idx] + n_nan_vals, no_data_vals = zip(*var_stats) + mean = np.sum(n_nan_vals) / (450 * len(paths)) + no_data = np.sum(no_data_vals) / len(paths) + + data[str(var_name)] = {"mean": np.float64(mean), "no_data": no_data} + + with open("nan_de23.json", "w") as fp: # statistics + json.dump(data, fp) + diff --git a/scripts/preprocessing/save_minicube.py b/scripts/preprocessing/save_minicube.py old mode 100644 new mode 100755 diff --git a/scripts/preprocessing/statistics_de23.py b/scripts/preprocessing/statistics_de23.py new file mode 100755 index 0000000..fa40d76 --- /dev/null +++ b/scripts/preprocessing/statistics_de23.py @@ -0,0 +1,113 @@ +from pathlib import Path +import xarray as xr +import numpy as np +import pandas as pd +import json +from tqdm import tqdm +import multiprocessing as mp +import os + + +temporal_variables = [ + "t2m_mean", + "pev_mean", + "slhf_mean", + "ssr_mean", + "sp_mean", + "sshf_mean", + "e_mean", + "tp_mean", + "t2m_min", + "pev_min", + "slhf_min", + "ssr_min", + "sp_min", + "sshf_min", + "e_min", + "tp_min", + "t2m_max", + "pev_max", + "slhf_max", + "ssr_max", + "sp_max", + "sshf_max", + "e_max", + "tp_max", +] + +spatiotemporal_variables = [ + "B02", "B03", "B04", "B05", "B06", "B07", "B8A","SCL", "cloudmask_en" +] + +spatial_variables = ["cop_dem"] + +def calculate_variable_statistics(data): + """Function to calculate statistics for each variable in a sample""" + + max_value = np.nanmax(data) + min_value = np.nanmin(data) + sum_var = np.nansum(data) + n_obs = np.count_nonzero(~np.isnan(data)) + return max_value, min_value, sum_var, n_obs + +def calculate_nan(data): + """Function to calculate statistics for each variable in a sample""" + n_nan = np.count_nonzero(np.isnan(data)) + no_data = 1 if n_nan == data.shape[0] else 0 + return n_nan, no_data + +def calculate_sample_statistics(file): + minicube = xr.open_dataset(file, engine="zarr").load() + return [calculate_variable_statistics(minicube[variable].values) for variable in temporal_variables] + +def process_samples_in_parallel(paths): + "Function to process samples using pdata = minicube[variable].valuesarallel processing" + num_cores = mp.cpu_count() + print("numbers of cores availables: ", num_cores) + pool = mp.Pool(processes=100) + results = list(tqdm(pool.imap(calculate_sample_statistics, paths), total=len(paths))) + pool.close() + pool.join() + return results + +def get_filepaths(metadata_file): + # load csv + df = pd.read_csv(metadata_file, delimiter=",")[["path", "check"]] + return df.loc[ (df["check"] == 0) , ["path"]].values.tolist() + + +if __name__ == "__main__": + basepath = "/Net/Groups/BGI/work_2/scratch/DeepExtremes/dx-minicubes" + + filepaths = get_filepaths("/Net/Groups/BGI/work_2/scratch/DeepExtremes/mc_earthnet.csv") + + paths = [Path(basepath + fp[0]) for fp in filepaths] + + print(paths[:10]) + print("len of the dataset: ", len(paths)) + + # Call the function to process samples in parallel + sample_statistics = process_samples_in_parallel(paths) + + # Transpose the results to get statistics for each variable + variable_statistics = list(map(list, zip(*sample_statistics))) + + # save the statistics for each variable + data = {} + for var_idx, var_stats in enumerate(variable_statistics): + var_name = (temporal_variables)[var_idx] + max_vals, min_vals, sum_vals, n_obs_vals = zip(*var_stats) + mean = np.sum(sum_vals) / np.sum(n_obs_vals) + data[str(var_name)] = {"mean": np.float64(mean), "min": np.float64(np.nanmin(min_vals)), "max": np.float64(np.nanmax(max_vals))} + + # for var_idx, var_stats in enumerate(variable_statistics): + # var_name = (spatiotemporal_variables + temporal_variables + spatial_variables)[var_idx] + # n_nan_vals, no_data_vals = zip(*var_stats) + # mean = np.sum(n_nan_vals) / (450 * len(paths)) + # no_data = np.sum(no_data_vals) / len(paths) + + # data[str(var_name)] = {"mean": np.float64(mean), "no_data": no_data} + + with open("statistics_de23.json", "w") as fp: # + json.dump(data, fp) + diff --git a/scripts/preprocessing/statistics_en23.py b/scripts/preprocessing/statistics_en23.py old mode 100644 new mode 100755 diff --git a/scripts/score_en21x_v2.py b/scripts/score_en21x_v2.py old mode 100644 new mode 100755