Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submission workflow #13

Merged
merged 6 commits into from
Nov 26, 2023
Merged

Submission workflow #13

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e1d6957
draft of checks to spat4cast_submit()
emmamendelsohn Nov 10, 2023
f23d69d
basic script to transfer from submissions to forecasts
emmamendelsohn Nov 10, 2023
049b75f
respond to comments from PR#13, including saving files in hive struct…
emmamendelsohn Nov 14, 2023
b18e1ff
move instead of copy submissions from top-level submissions bucket
emmamendelsohn Nov 14, 2023
69a3564
somewhat roundabout way to prevent copying directory name into hive s…
emmamendelsohn Nov 14, 2023
2b3f09e
Update spat4cast_submit.R
cboettig Nov 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions R/spat4cast_submit.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,22 @@

spat4cast_submit <- function(dir, theme = "lai_recovery") {
# upload anonymously to submissions bucket
spat4cast_submit <- function(dir, theme = "lai_recovery", fire = "august_complex") {

submitted_files <- list.files(dir, full = TRUE)

# Check file type
assertthat::assert_that(n_distinct(tools::file_ext(submitted_files)) == 1)
assertthat::assert_that(unique(tools::file_ext(submitted_files)) == "tif", msg = "Error: The file extension in the directory is not 'tif'. Please check the file format.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great. 🎉

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, we probably want a dictionary of valid site_id names (ideally this will come from some spatial vector table we give people with site_id , date of fire/disturbance, and the polygon geometry).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, added some documentation and set the function to enforce valid site_id and variable names:
https://github.com/eco4cast/modis-lai-forecast/blob/submission-workflow/R/spat4cast_submit.R#L5-L20


# Check bounding box for files
## can we use the `fire` argument to pull a template target file for comparison?
## if we are to enforce these attributes, then we should modify `ingest_planetary_data` to remove dy, dx, dt, srs arguments (and have them enforced according to the fire of intered)
res(rast(submitted_files))
origin(rast(submitted_files))
ext(rast(submitted_files))

## Upload anonymously to submissions bucket
minioclient::mc_alias_set("efi", "data.ecoforecast.org", "", "")
minioclient::mc_cp(dir,
paste0("efi/spat4cast-submissions/", theme),
paste0("efi/spat4cast-submissions/", theme, "/", fire), # should we have separate buckets for each fire?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call, that's probably easiest. paralleling the existing conventions in efi, I think we should refer to theme as the variable and the fire location as the site_id, and let's express this in hive format, so maybe something like:

glue("efi/spat4cast-submissions/variable={theme}/site_id={fire}")

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh somewhere in here we need a model_id={model_id} as well, since we will have different models (from the same or different teams of people) forecasting the same site_id / reference_datetime etc. (I guess we currently have that in the filenames but might help to have it explicitly on the path).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you suggest validating model_id? would we need to provide guidance on how to identify the team and model type?

recursive = TRUE)
}
5 changes: 5 additions & 0 deletions process_submissions.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## Send from submissions to forecasts
minioclient::mc_alias_set("efi", "data.ecoforecast.org", "", "")
minioclient::mc_cp(from = paste0("efi/spat4cast-submissions/", theme),
to = paste0("efi/spat4cast-forecasts/", theme),
recursive = TRUE)