Code to generate EZfate data

[JamiePringle]

Dear EZfate users -- In the next several messages I will post the tool change that I use to make the data that I serve with EZfate. This code is not uploaded to the repository because it is not designed for general use. It is here so you can see what I have done, and so you can adapt it if you wish. But read the code carefully, and think if you really want to start with such a specialized code.

This toolchain runs well on a machine with 256Gb of RAM and 32 cores. Fewer cores would be ok, but less memory would require some adaptation.

Note that I do not recommend you grab this code blindly and use it for your problem. It is uses the old version 2.2 of oceanParcels, and is not set up to be a general purpose tool chain. It is very strongly designed for a global multi-year project, and so is focused on efficiency in both computer time and disk storage. In general, for a more regional problem, I would keep the particle tracks in a full latitude and longitude format, and not change the locations to a grid index. Also, early on, we trimmed the periodic padding points from the Mercator GLORYS grid. This was a mistake, and if I were to re-write the code, I would not do this again. Also, the file names are based on the old Matlab date numbering scheme, which is awkward...

The first thing you need to do to use this code is download the global circulation data from Mercator. The following code does this. Note for EZfate alone, you only need to get horizontal and vertical velocity -- I get the other fields for other projects. Also, you will need to make a Copernicus ocean account that allows you to download the C-grid data (the A-grid data will not work). In this code, I have redacted my credentials -- you will need to obtain your own from Copernicus Ocean.

To use this code, set up your Copernicus Ocean account, and then set the dates in the python file to download the data you need. Note that this code will skip files if it can't read them and crash if the network blinks. You will need to keep running this code until it gets all the files it needs. Just keep running it. It will check what it has, and try to get what it does not.

Data Download Code: read_allKinds_data_toShare_noPasswords.py

After the files are downloaded, run the following two codes to move them to permanent storage sorted by years, and then the second code which adds symbolic links from a single directory with links to all files in all years to the actual data files. This may seem awkward and convoluted, but allows you to have storage spread across multiple disk groups.

moveData_DONT_rechunk.py and
makeSwitchboard.py

Before you start running any of the code below, make in the directory you are running in the directories dataPaths, dataPathsTemp and dataMatrices. After you are done, you can delete the contents of dataPathsTemp. The locations of the final R connectivity matrices is set in yieldConMatrix.py, discussed below. These directories can be very large -- on my machines they are actually symbolic links to directories on larger files systems.

Thank you to @sophia-wright-blue for pushing me to post this code

[JamiePringle]

The script runSomeYear_Rip.com runs the python code 01_makeDispersalData_fromHabitat_RIP_splitYear.py which sets up the particle tracking and then calls trackPointsScript.py which does the actual particle tracking. These run only track particles for the Eastern or Western Hemisphere, for a given particle depth and vertical behavior (constant depth, or variable depth). All of these parameters are set in 01_makeDispersalData_fromHabitat_RIP_splitYear.py. Which year(s) are to be run is set in runSomeYear_Rip.com

You must run a full year and both hemispheres before continuing on to the next set of codes. This will take a while.

01_makeDispersalData_fromHabitat_RIP_splitYear.py

trackPointsScript.py

I can't upload *.com files, so here is the script runSomeYear_Rip.com for you to copy

#!/bin/bash


#for year in {2016..2022}
for year in 2024
do
    #python warm_up_cache.py $year > warm_up_log.txt &
    python 01_makeDispersalData_fromHabitat_RIP_splitYear.py $year
done

[JamiePringle]

The next file, 02_fix_date_attributes.py just reads in the data files and fixes the date attribute on the zarr file. It will check to see if the dates make any sense, and will not re-fix an already fixed file. You need to specify in the code which depth and vertical behavior directory you want to process.

02_fix_date_attributes.py

[JamiePringle]

The next file, 03_make_goodHabitat_files.py needs to be run only once for each launch depth and each launch habitat. It DOES NOT need to be run for each year or vertical behavior. It makes a bunch of geographic sub-regions (e.g. "theAmericas," see the EZfate documentation) and stores which drifter launch locations belong to each region.

03_make_goodHabitat_files.py

[JamiePringle]

The script 04_makeConnectivity_byMonth_andYear.py and the associated makeConnectivityModule.py is the heart of the connectivity calculation. It takes in the calculated Lagrangian pathways and calculates the connectivity matrices for a series of drift durations, months and years.

Before running the code, you must specify in 04_makeConnectivity_byMonth_andYear.py what release depth, vertical behavior and years you are creating the connectivity matrices for. By default, it will create matrices for durations from 2 days to 60 days with an interval of 2 days.

Note, (and this is one of the things that would be fixed in a more polished code), you must run the script TWICE! First with the option makeStartDay=True, then again with makeStartDay=False. Alas, I wrote this before I understood how xarray interacted with multiprocessing.

04_makeConnectivity_byMonth_andYear.py

makeConnectivityModule.py

[JamiePringle]

Now once you have created all your years of connectivity matrices, it is useful to have climatological connectivity matrices. These are created by 05_makeClimatological_matrices_OneMonth.py. Before running it, you must edit the code to specify what depth you are processing and which vertical behavior style you are processing. It will loop over all drift durations on its own, if you have not changed that above. You MUST also define which years you are calculating the climatology over by altering the values of the variables minYear and maxYear. On my machine, it is useful to be able to run this code in parallel, so I run it with this script:

#!/bin/bash -f

echo Starting parallel runs, check parOut for log files 
python 05_makeClimatological_matrices_OneMonth.py 1 > parOut/1.txt &
python 05_makeClimatological_matrices_OneMonth.py 2 > parOut/2.txt &
python 05_makeClimatological_matrices_OneMonth.py 3 > parOut/3.txt &
python 05_makeClimatological_matrices_OneMonth.py 4 > parOut/4.txt & 
python 05_makeClimatological_matrices_OneMonth.py 5 > parOut/5.txt & 
python 05_makeClimatological_matrices_OneMonth.py 6 > parOut/6.txt & 
python 05_makeClimatological_matrices_OneMonth.py 7 > parOut/7.txt & 
python 05_makeClimatological_matrices_OneMonth.py 8 > parOut/8.txt & 
python 05_makeClimatological_matrices_OneMonth.py 9 > parOut/9.txt & 
python 05_makeClimatological_matrices_OneMonth.py 10 > parOut/10.txt & 
python 05_makeClimatological_matrices_OneMonth.py 11 > parOut/11.txt & 
python 05_makeClimatological_matrices_OneMonth.py 12 > parOut/12.txt &

echo all started, calling wait
wait
echo all done

Note well -- I have a relatively large machine, your mileage may vary.

05_makeClimatological_matrices_OneMonth.py

[JamiePringle]

Finally, the audience of much of this work are used to working in R, so I want to convert the matrices to a more R friendly format. To do this I write a python code which uses R library reticulate to call a python generator which yields all the of the connectivity matrices it can find. It will traverse the entire directory of zarr connectivity matrices, check if the R versions exists, and only converts the file if it has not yet been converted. So you can just run this over and over, until there are no more files to convert. This is useful, because you can start converting files before the 04* and 05* steps above have finished. Since this conversion process is relatively lightweight, starting on conversion to R before all the connectivity matrices are created can save you a great deal of time.

The location where the python and R matrices are stored are defined in yieldConMatrix.py -- note that the zarrDataPath defined here is the same directory as the dataMatrices directory, but I give it a full pathname because of how I managed storage for my runs.

The R file is saveZarrMats_2_R.R and the python is yieldConMatrix.py. To use this code, enter the R environment and source saveZarrMats_2_R.R.

I STRONGLY suggest using conda to install both R and python in the same environment -- don't try to mash together an existing R (or Rstudio) and python set of installations, unless you want to get deep into the weeds of how shared libraries work on your machine. To make setting up a conda environment with its own R installation easier I have included the file RandPython_May2023.yml. Instructions on how to use this *.yml file can be found at https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file .