gdas_marinebmat job failure

[JessicaMeixner-NOAA]

@jiandewang and @ChristianBoyer-NOAA were running a test on WCOSS2 with current develop plus NOAA-EMC/global-workflow#4258 and ran into the issue:

nid004062.cactus.wcoss2.ncep.noaa.gov 63:
FATAL from PE 63: fms_io(get_file_unit): max_files_r exceeded, increase it via fms_io_nml

Jiande's log file can be found here: /lfs/h2/emc/ptmp/jiande.wang/work-v17-test/comroot/v17-test/logs/2024051512/gdas_marinebmat.log.0 Christian independently also go the same error.

They both used the https://github.com/NOAA-EMC/global-workflow/blob/develop/dev/ci/cases/gfsv17/retrov17_stream2.yaml (with slight updates for running from their account). This is the same as what we are running in retrov17 but a newer hash. This gdas app hash update NOAA-EMC/global-workflow#4236 is not included in the retros, so there could be something in that PR.

@guillaumevernieres FYI

[JessicaMeixner-NOAA]

@guillaumevernieres - Any insight on this?

[guillaumevernieres]

@guillaumevernieres - Any insight on this?

@JessicaMeixner-NOAA , It's on my to do list but I haven't had time to look at the log yet.

[JessicaMeixner-NOAA]

The log is going to go away soon since it's on ptmp. I'm worried if we update the code things might fail - I'm not sure where this is coming from or if the relevant updates will be included when we get other updates or not ... @CatherineThomas-NOAA

[guillaumevernieres]

The log is going to go away soon since it's on ptmp. I'm worried if we update the code things might fail - I'm not sure where this is coming from or if the relevant updates will be included when we get other updates or not ... @CatherineThomas-NOAA

@JessicaMeixner-NOAA , @CatherineThomas-NOAA : @apchoiCMD is going to test today.

[apchoiCMD]

The log is going to go away soon since it's on ptmp. I'm worried if we update the code things might fail - I'm not sure where this is coming from or if the relevant updates will be included when we get other updates or not ... @CatherineThomas-NOAA

@JessicaMeixner-NOAA , @CatherineThomas-NOAA : @apchoiCMD is going to test today.

Kept the log in my place, I will set up for this in this afternoon.

[CatherineThomas-NOAA]

I'm worried if we update the code things might fail - I'm not sure where this is coming from or if the relevant updates will be included when we get other updates or not ... @CatherineThomas-NOAA

I ran a test on Gaea-C6 that included only the changes directly related to NOAA-EMC/global-workflow#4236 and I didn't have any issues with the bmat job. However, my test did not include the Weekly JEDI hash update from 7cb324f that was also merged to g-w develop when updating the GDASApp hash for NOAA-EMC/global-workflow#4236. That GDASApp hash update includes updated hashes for SOCA and OOPS, which could be a potential source of the error. This could also be a WCOSS vs Gaea issue, but I'm not sure at the moment.

[JessicaMeixner-NOAA]

Did you run a full res experiment or just CI?

[CatherineThomas-NOAA]

Ah yes, it was the low res CI test so if it's an issue of reading too many files, the CI won't catch that with only a couple ensemble members.

[guillaumevernieres]

The quick fix is to update the number of files we can open, use 200 instead of 100 here:

GDASApp/parm/marine/fms/input.nml.j2

Line 15 in 54dbb71

max_files_w=100

this should get you going while we figure out why we are opening more files suddenly.

[JessicaMeixner-NOAA]

I ran a test with https://github.com/NOAA-EMC/GDASApp/tree/release/gfs.v17 and did not get the issue. That means this issue isn't urgent for GFS.

[apchoiCMD]

Thanks @JessicaMeixner-NOAA for the test-