Current version of performance notebooks#56

Open

jsignell wants to merge 373 commits intoprimarykeyv2from

performance-notebooks

This pull request is big! We're only showing the most recent 250 commits

Commits on Mar 24, 2024

so much configuration
ranchodeluxe
committed
change maap container
ranchodeluxe
committed

Commits on Mar 25, 2024

build.sh
ranchodeluxe
committed
use vanilla base images
ranchodeluxe
committed
pushd basedir
ranchodeluxe
committed
change version of submitJob Docker image
ranchodeluxe
committed
need to install MAAP api
ranchodeluxe
committed
more docker trickery with vanilla image
ranchodeluxe
committed
more docker trickery with vanilla image
ranchodeluxe
committed
source this way
ranchodeluxe
committed
try parameter expansion in bash
ranchodeluxe
committed
parametrize algo_name too
ranchodeluxe
committed
w.e.
ranchodeluxe
committed
try without docker container
ranchodeluxe
committed
try with new token
ranchodeluxe
committed
try with export
ranchodeluxe
committed
try this
ranchodeluxe
committed
remove comments
ranchodeluxe
committed
try straight shell
ranchodeluxe
committed
events
ranchodeluxe
committed
python version
ranchodeluxe
committed
try python3.10
ranchodeluxe
committed
no sudo
ranchodeluxe
committed
run work
ranchodeluxe
committed
one more time
ranchodeluxe
committed
out of luck
ranchodeluxe
committed
this
ranchodeluxe
committed
don't use docker container
ranchodeluxe
committed
-w option
ranchodeluxe
committed
add MAAP conf
ranchodeluxe
committed
with protected off
ranchodeluxe
committed
final iteration
ranchodeluxe
committed
clean it up
ranchodeluxe
committed
more cleanup
ranchodeluxe
committed
rebuid images and scalene args
ranchodeluxe
committed
let's build images again :disappoint:
ranchodeluxe
committed

Commits on Mar 26, 2024

have wget send back download path in DataCheckUpdate
ranchodeluxe
committed
tweak logging
ranchodeluxe
committed
read_region needs to use fsspec
ranchodeluxe
committed
clean up FireRunByRegionAndT.py
ranchodeluxe
committed
all_fires_filepath fix
ranchodeluxe
committed
add uploading feedback
ranchodeluxe
committed
revert
ranchodeluxe
committed
more logging
ranchodeluxe
committed
return output filepaths
ranchodeluxe
committed
change region/year order in path
ranchodeluxe
committed
Merge branch 'gc/pipelines/additions' into 'pipelines'
ranchodeluxe
committed
Update notebooks and add numbers
jsignell
committed
Don't worry if only one input file is available
jsignell
committed
moar pipeline changes

ranchodeluxe
authored and
jsignell
committed
Merge branch 'gc/moar/pipelines' into 'pipelines'
jsignell
committed
Merge branch 'resilient-to-inputs' into 'pipelines'
jsignell
committed

Commits on Mar 27, 2024

dask coordinator
ranchodeluxe
committed
just four
ranchodeluxe
committed
change call sites
ranchodeluxe
committed
more changes
ranchodeluxe
committed
max workers
ranchodeluxe
committed
make example CONUS
ranchodeluxe
committed
datetime
ranchodeluxe
committed
allow me to kick off work from dask branch
ranchodeluxe
committed
simplify outdir
ranchodeluxe
committed
Merge branch 'gc/dask_coordinator' into 'pipelines'
ranchodeluxe
committed
coverage stuff
ranchodeluxe
committed
all the coverages
ranchodeluxe
committed
try this for filepaths
ranchodeluxe
committed
Merge branch 'gc/pytest' of https://repo.maap-project.org/gcorradini/fireatlas_nrt into gc/pytest
ranchodeluxe
committed
do local
ranchodeluxe
committed
filepath examples
ranchodeluxe
committed
update remove static sources
ranchodeluxe
committed
Update the output bucket
jsignell
committed
Merge branch 'gc/pytest' into pipelines
ranchodeluxe
committed
Merge branch 'pipelines' into 'primarykeyv2'
ranchodeluxe
committed

Commits on Mar 28, 2024

Commits on Mar 29, 2024

take suggestions
ranchodeluxe
committed
add package name to all imports
ranchodeluxe
committed
reorg maap scripts and add docs
ranchodeluxe
committed
naming matters
ranchodeluxe
committed
changes
ranchodeluxe
committed
Merge branch 'gc/reorg' of https://repo.maap-project.org/gcorradini/fireatlas_nrt into gc/reorg
ranchodeluxe
committed
whatever
ranchodeluxe
committed

Commits on Apr 1, 2024

final changes
ranchodeluxe
committed
notebook updates
ranchodeluxe
committed
Merge branch 'gc/reorg' into 'primarykeyv2'
ranchodeluxe
committed

Commits on Apr 2, 2024

example
ranchodeluxe
committed
github
ranchodeluxe
committed
operation
ranchodeluxe
committed
10 days
ranchodeluxe
committed
add workflow_dispatch
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Merge branch 'primarykeyv2' into gc/github_actions
ranchodeluxe
committed
force new setuptools version
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Merge branch 'primarykeyv2' into gc/github_actions
ranchodeluxe
committed
more
ranchodeluxe
committed
do it again
ranchodeluxe
committed
try this
ranchodeluxe
committed
change branches
ranchodeluxe
committed
Add notebooks that run a year and make it so you can pick up and run a t
jsignell
committed
Pick up and run Fire_Forward
jsignell
committed

Commits on Apr 5, 2024

Commits on Apr 6, 2024

whoops
ranchodeluxe
committed
Merge branch 'gc/dps/changes' into 'primarykeyv2'
ranchodeluxe
committed

Commits on Apr 7, 2024

change fireatlas copy
ranchodeluxe
committed
running log
ranchodeluxe
committed
correct location of logging
ranchodeluxe
committed

Commits on Apr 8, 2024

changes
ranchodeluxe
committed
Merge branch 'gc/running_log' into 'primarykeyv2'
ranchodeluxe
committed
handle merge conflicts
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
make it so we don't have to pass tst/ted for anything
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on Apr 9, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on Apr 10, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Fix datetime
jsignell
committed
Merge branch 'js/fix-datetime' into 'primarykeyv2'
jsignell
committed

Commits on Apr 11, 2024

parallelize uploads
ranchodeluxe
committed
this works
ranchodeluxe
committed
delay the results proper to call .compute
ranchodeluxe
committed
two dags?
ranchodeluxe
committed
dask is stupid
ranchodeluxe
committed
dask is stupid
ranchodeluxe
committed
try this
ranchodeluxe
committed
Merge branch 'gc/uploads_dask' into 'primarykeyv2'
ranchodeluxe
committed
more docs
ranchodeluxe
committed

Commits on Apr 12, 2024

add back files
ranchodeluxe
committed
fix bug
ranchodeluxe
committed
stupid dask again
ranchodeluxe
committed
two computes
ranchodeluxe
committed

Commits on Apr 16, 2024

small changes to run independently
ranchodeluxe
committed
tweaks
ranchodeluxe
committed

Commits on Apr 17, 2024

so difficult
ranchodeluxe
committed
stuff
ranchodeluxe
committed

Commits on Apr 25, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on Apr 26, 2024

force to TZ to EST/EDT
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
break into smaller dags
ranchodeluxe
committed
export TZ=Etc/UTC
ranchodeluxe
committed
remove scheduler for data-update
ranchodeluxe
committed

Commits on Apr 27, 2024

try this
ranchodeluxe
committed

Commits on May 2, 2024

fix comments
ranchodeluxe
committed
Merge pull request #3 from Earth-Information-System/gc/smallfix
ranchodeluxe
authored

Commits on May 8, 2024

updated tst and ted to be compatible with Julias updates
lithal1
committed
adding in the rest of the changes from that pairing session
lithal1
committed

Commits on May 9, 2024

path to preprocessed VIIRS files not correct bc fs should have been read from settings
lithal1
committed

Commits on May 10, 2024

Merge pull request #6 from Earth-Information-System/fixRunConstrain
lisamr
authored

Commits on May 13, 2024

manually resolve merge conflicts
ranchodeluxe
committed
Merge branch 'primarykeyv2' of github.com:Earth-Information-System/fireatlas into primarykeyv2
ranchodeluxe
committed
set log level to info
ranchodeluxe
committed
change DPS image algo pull
ranchodeluxe
committed
change checkout location
ranchodeluxe
committed
resolve merge conflicts
ranchodeluxe
committed

Commits on May 14, 2024

write func to find files that need preprocessing
ranchodeluxe
committed
check region-t preprocessed existence
ranchodeluxe
committed
Merge pull request #11 from Earth-Information-System/gc/needing_region_t_preprocessing
ranchodeluxe
authored
start
ranchodeluxe
committed
add error handling for grace and profit
ranchodeluxe
committed

Commits on May 16, 2024

Fix Issues reported by Yang (#4)
jsignell
authored
change RunByRegionAndT
ranchodeluxe
committed
fix bug
ranchodeluxe
committed
unhashable type
ranchodeluxe
committed
Merge pull request #15 from Earth-Information-System/gc/inspect_empty_regiont_outputs
ranchodeluxe
authored
veda s3 copy changes
ranchodeluxe
committed

Commits on May 17, 2024

tune down runs
ranchodeluxe
committed
Merge pull request #16 from Earth-Information-System/gc/tunedown_scheduler
ranchodeluxe
authored
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Merge branch 'primarykeyv2' of github.com:Earth-Information-System/fireatlas into primarykeyv2
ranchodeluxe
committed
Merge pull request #18 from Earth-Information-System/gc/copy_to_veda
ranchodeluxe
authored
run last few
ranchodeluxe
committed
Merge pull request #12 from Earth-Information-System/gc/pydantic_overrides
ranchodeluxe
authored
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
Merge branch 'primarykeyv2' of github.com:Earth-Information-System/fireatlas into primarykeyv2
ranchodeluxe
committed

Commits on May 18, 2024

Merge branch 'primarykeyv2' into gc/run_last_few
ranchodeluxe
committed
register new job
ranchodeluxe
committed
sort it
ranchodeluxe
committed
Merge pull request #25 from Earth-Information-System/gc/run_last_few
ranchodeluxe
authored

Commits on May 21, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed
small fixes to get snapshots copying to veda
ranchodeluxe
committed
just reprocess
ranchodeluxe
committed
logger.warn
ranchodeluxe
committed
Merge pull request #27 from Earth-Information-System/gc/fixes_for_ingest
ranchodeluxe
authored
only take latest ted snapshots
ranchodeluxe
committed
new processing power
ranchodeluxe
committed

Commits on May 22, 2024

do not do anything
ranchodeluxe
committed
raise KeyError
ranchodeluxe
committed
remove existing combine LF flows
ranchodeluxe
committed
Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on May 23, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on May 24, 2024

use t instad of ted
ranchodeluxe
committed
use t instad of ted
ranchodeluxe
committed
Merge branch 'primarykeyv2' into gc/remove_combine_lf
ranchodeluxe
committed
seconda pass
ranchodeluxe
committed
vectorized
ranchodeluxe
authored
save combined
ranchodeluxe
committed
Update postprocess.py
ranchodeluxe
authored
make sure it works
ranchodeluxe
committed

Commits on May 28, 2024

Commits on May 29, 2024

Merge branch 'conus-dps' into primarykeyv2
ranchodeluxe
committed

Commits on May 31, 2024

Change to check monthly first and use dask futures (#39)

authored

Commits on Jun 4, 2024

back to 3.1.4
ranchodeluxe
committed

Commits on Jun 5, 2024

dump out s3fs version
ranchodeluxe
committed
try it again
ranchodeluxe
committed
remove scalene temmporarily
ranchodeluxe
committed
Merge pull request #51 from Earth-Information-System/gc/remove_scalene_temporarily
ranchodeluxe
authored
If there are no new pixels carry on (#47)
jsignell
authored
add python
ranchodeluxe
committed
Merge branch 'primarykeyv2' of github.com:Earth-Information-System/fireatlas into primarykeyv2
ranchodeluxe
committed

Commits on Jun 6, 2024

250GB
ranchodeluxe
committed
Fix for postprocess where `t_saved=None` on runs from scratch (#52)

ranchodeluxe
and
ranchodeluxe
authored

Commits on Jun 12, 2024

Current version of performance notebooks
jsignell
committed

Commits on Jul 2, 2024

Now that run is complete, update notebooks
jsignell
committed