Bugfix/leadtime multilandfall

[SorooshMani-NOAA]

Fixing a couple of issues related to picking up the leadtime dates for multi-landfall storms

[SorooshMani-NOAA]

Hi @FariborzDaneshvar-NOAA this includes an old update that was "luckily" on Hercules, so we could run all storms on Hercules, and then also a fix for the issue @WPringle uncovered about landfall order - in specific that impacted Matthew 2016, but still it is worth adding the fix! Please let me know if you see any issues with my changes, I haven't been spending a lot of time on this repo recently. Thanks!

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA Sorry for the late response. It only gets the first landfall, correct?
It looks good to me. Do you want me to run the workflow and test it before merging?

[SorooshMani-NOAA]

There's no rush for merging this ... whenever you get to run a new ensemble, if you can please test this and make sure first landfall is being simulated then I'll merge. We have to later add an input parameter in yaml for selecting landfall, but for now this is just a bugfix!

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA which storm do you recommend to test? Matthew or Irma?

[SorooshMani-NOAA]

If you could verify both work fine it'd be great, thanks!

[FariborzDaneshvar-NOAA]

I ran the workflow (without this fix) for Irma 2017, and Matthew 2016. I got this error for getting Matthew hurricane info.:

 2025-05-15:11:06:32,39 INFO     [hurricane_data.py:159] Fetching hurricane info...
Traceback (most recent call last):
  File "/work2/noaa/nos-surge/daneshva/miniconda3/envs/stormworkflow/bin/hurricane_data", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/work2/noaa/nosofs/daneshva/ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py", line 456, in cli
    main(args)
  File "/work2/noaa/nosofs/daneshva/ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py", line 168, in main
    prescribed = trackstart_from_file(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/nosofs/daneshva/ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py", line 70, in trackstart_from_file
    return storm_all_times[leadtime].item()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Timestamp' object has no attribute 'item'

[SorooshMani-NOAA]

Can you double check to see if you have

ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py

Lines 65 to 69 in 630a1e6

	storm_all_times = leadtime_table.loc[[nhc_code.lower()]].dropna()
	if len(storm_all_times) > 1:
	storm_all_times = storm_all_times.iloc[[0]]
	if leadtime not in storm_all_times:
	return None

and not

ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py

Lines 64 to 68 in 1648dd7

	storm_all_times = leadtime_table.loc[[nhc_code.lower()]].dropna()
	if len(storm_all_times) > 1:
	storm_all_times = storm_all_times.iloc[0]
	if leadtime not in storm_all_times:
	return None

[FariborzDaneshvar-NOAA]

Can you double check to see if you have

ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py

Lines 65 to 69 in 630a1e6

	storm_all_times = leadtime_table.loc[[nhc_code.lower()]].dropna()
	if len(storm_all_times) > 1:
	storm_all_times = storm_all_times.iloc[[0]]
	if leadtime not in storm_all_times:
	return None

and not

ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py

Lines 64 to 68 in 1648dd7

	storm_all_times = leadtime_table.loc[[nhc_code.lower()]].dropna()
	if len(storm_all_times) > 1:
	storm_all_times = storm_all_times.iloc[0]
	if leadtime not in storm_all_times:
	return None

That's right! I had the wrong one! and it worked after this fix iloc[[0]] instead of iloc[0]

[FariborzDaneshvar-NOAA]

Here are the dates from dates.csv files for:

• Matthew old setup (without sorting):

  0,2016-10-03 18:00:00
  1,2016-10-10 06:00:00
  2,2016-10-05 18:00:00

• Matthew new setup:

  0,2016-10-03 18:00:00
  1,2016-10-10 06:00:00
  2,2016-10-05 18:00:00

• Irma old setup (without sorting):

  0,2017-09-06 18:00:00
  1,2017-09-13 12:00:00
  2,2017-09-08 18:00:00

• Irma new setup:

  0,2017-09-06 18:00:00
  1,2017-09-13 12:00:00
  2,2017-09-08 18:00:00

They are the same for both setups. I also checked /nhc_track/hurricane-track.dat files and they were the same for two sets of runs.
@SorooshMani-NOAA Is there anything else that you want me to check?

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA I think Irma has a single landfall in the leadtime file. For Matthew this shouldn't be the case ... let's talk about it.

Actually when I get a chance, I probably need to just add a unit test for this

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA I added the unit test and a sample leadtime file. Note that for storm C, the order of landfalls are flipped, so it needs sorting. No rush, please at your convenience checkout this test and confirm that removing sort part of the new code causes the test to fail. Thanks!

You can run the test by installing pytest and then calling pytest -k leadtime while you're in the repo main directory. Then you can remove the sort_values call and see it fails (meaning the new addition fixes picking up the first landfall). Also confirm the landfall times in the newly fake leadtime file I added under tests/data/refs

[SorooshMani-NOAA]

I just noticed the CI test failed ... please wait for the CI test to pass, then try it on your side!!

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA you can look check the code now (whenever you have time)

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA I ran pytest -k leadtime command in the ondemand-storm-directory and this is what I got:

============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.11.12, pytest-8.3.5, pluggy-1.6.0
rootdir: /lustre/Fariborz.Daneshvar/scratch/ondemand-storm-workflow
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 8 items / 7 deselected / 1 selected

tests/test_prep_funcs.py .                                                                                                                                                                                [100%]

=============================================================================================== warnings summary ================================================================================================
../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/geopandas/_compat.py:7
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/geopandas/_compat.py:7: DeprecationWarning: The 'shapely.geos' module is deprecated, and will be removed in a future version. All attributes of 'shapely.geos' are available directly from the top-level 'shapely' namespace (since shapely 2.0.0).
    import shapely.geos

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:9
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:9: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    persistent = auto()

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:10
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:10: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    regression_penny_2023_with_smoothing = auto()

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:12
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:12: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    regression_penny_2023_no_smoothing = auto()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== 1 passed, 7 deselected, 4 warnings in 4.08s ==================================================================================

Should I complete the PR?

[FariborzDaneshvar-NOAA]

I then removed .sort_values([0]):

ondemand-storm-workflow/stormworkflow/prep/hurricane_data.py

Line 60 in bb57604

).set_index('ALnumber').sort_values([0])

And ran the test again. This time the test failed as expected:

============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.11.12, pytest-8.3.5, pluggy-1.6.0
rootdir: /lustre/Fariborz.Daneshvar/scratch/ondemand-storm-workflow
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 8 items / 7 deselected / 1 selected

tests/test_prep_funcs.py F                                                                                                                                                                                [100%]

=================================================================================================== FAILURES ====================================================================================================
______________________________________________________________________________________________ test_leadtime_pick _______________________________________________________________________________________________

leadtime_file = PosixPath('/lustre/Fariborz.Daneshvar/scratch/ondemand-storm-workflow/tests/data/refs/leadtime.json')

    def test_leadtime_pick(leadtime_file):

        # Always picks first
        assert datetime(2049, 10, 3, 6) == trackstart_from_file(
            leadtime_file=leadtime_file,
            nhc_code="al082049",
            leadtime=48)

>       assert datetime(2050, 9, 28, 0) == trackstart_from_file(
            leadtime_file=leadtime_file,
            nhc_code="al142050",
            leadtime=24)
E       AssertionError: assert datetime.datetime(2050, 9, 28, 0, 0) == Timestamp('2050-09-30 00:00:00')
E        +  where datetime.datetime(2050, 9, 28, 0, 0) = datetime(2050, 9, 28, 0)
E        +  and   Timestamp('2050-09-30 00:00:00') = trackstart_from_file(leadtime_file=PosixPath('/lustre/Fariborz.Daneshvar/scratch/ondemand-storm-workflow/tests/data/refs/leadtime.json'), nhc_code='al142050', leadtime=24)

tests/test_prep_funcs.py:15: AssertionError
=============================================================================================== warnings summary ================================================================================================
../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/geopandas/_compat.py:7
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/geopandas/_compat.py:7: DeprecationWarning: The 'shapely.geos' module is deprecated, and will be removed in a future version. All attributes of 'shapely.geos' are available directly from the top-level 'shapely' namespace (since shapely 2.0.0).
    import shapely.geos

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:9
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:9: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    persistent = auto()

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:10
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:10: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    regression_penny_2023_with_smoothing = auto()

../../miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:12
  /lustre/Fariborz.Daneshvar/miniconda3/envs/stormworkflow_landfall/lib/python3.11/site-packages/stormevents/nhc/const.py:12: DeprecationWarning: In 3.13 the default `auto()`/`_generate_next_value_` will require all values to be sortable and support adding +1
  and the value returned will be the largest value in the enum incremented by 1
    regression_penny_2023_no_smoothing = auto()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================ short test summary info ============================================================================================
FAILED tests/test_prep_funcs.py::test_leadtime_pick - AssertionError: assert datetime.datetime(2050, 9, 28, 0, 0) == Timestamp('2050-09-30 00:00:00')
================================================================================== 1 failed, 7 deselected, 4 warnings in 4.31s ==================================================================================