Skip to content

Parallel download of GOES images#7

Open
ninarobbins wants to merge 19 commits intoGeet-George:mainfrom
ninarobbins:parallel_download
Open

Parallel download of GOES images#7
ninarobbins wants to merge 19 commits intoGeet-George:mainfrom
ninarobbins:parallel_download

Conversation

@ninarobbins
Copy link
Contributor

This is a pull request to add the scripts used to download GOES images taking advantage of parallelization. The way it is structured is in 3 python scripts and 3 shell scripts which run each of these python scripts individually.

  • First, one needs a text file with dates as its rows in the format YYYYMMDD.
  • Then the script run_mapper_goes16.sh is run as sbatch run_mapper_goes16.sh /path/to/dates_file.txt. This will generate the ".json" mapper files for all dates and store them in the path specified in mapper_goes16.py. These contain the mapper to the entire GOES data for that date.
  • Then, we run sbatch run_save_nc_goes16.sh /path/to/dates_file.txt. This runs the file save_nc_goes16.py, which acesses the data through the mappers we saved, selects a subset in space, and saves that subset to a netCDF file for each day into the folder specified in the python file.
  • Finally, we can plot the GOES-16 data every 10 mins for each day by running sbatch run_plot_goes16.sh /path/to/dates_file.txt, which runs the file plot_goes16.py, and saves each image in the folder defined in the python file.

This all works fine, but it might not be the most optimized or concise way to do this. Generating the mappers is the most time consuming task by far, which is why here it is separated into a different process (you don't have to generate them every time you want a different subset/plot, you only need to run it once for a day and then you can use it to access all the data). Here the paths are hard coded into the scripts, which probably should be changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant