binarytile.x and resolution: Should we stock binary tile files?

[mathomp4]

This is from a discussion between @biljanaorescanin and @mathomp4 (along with @tclune and @sshakoor1)

Namely, should we be providing binary versions of the Pfaffstetter tile files generated by @biljanaorescanin ? At run-time, we run binarytile.x when we link our BCs and this can be significant at high res. To wit:

Resolution	Time (s)
CF0012	1.09
CF0024	1.20
CF0048	1.40
CF0090	1.91
CF0180	5.31
CF0360	10.73
CF0720	20.46
CF1120	36.24
CF1440	55.18
CF2880	189.11
CF5760	702.50

Now, up to even C720, it's only 20 seconds. Given the vagaries of Discover or NAS disks, this can be nigh-ignored.

But it does seem to increase pretty good at very-high-res. 3 minutes at 2880 and 12 minutes at 5760.

So, perhaps, should we provide users with "pre-made" binary tile files at a certain resolution?

The one caveat is there is no guarantee that binaries made by Intel would be usable with GNU and vice versa. Then again, maybe we only need to include files from our production compiler?

I'll mention @gmao-rreichle as well as if we decide to do this, we'd also need to think about how to organize it.

[gmao-rreichle]

Don't we already have nc4 tile files, at least for the latest bcs version? How long does it take to read those? I don't think we want binary. Or maybe I'm missing something

[tclune]

@mathomp4 While portability cannot be guaranteed, I strongly suspect that the files will work for both compilers. Probably even NAG. Most compilers adopted a convention about how the records are separated. I think some might use number of words vs number of bytes. But I think intel and gfortran made the same choice. (ANd we don't have big endian machines to worry about these days.)

[tclune]

Don't we already have nc4 tile files, at least for the latest bcs version? How long does it take to read those? I don't think we want binary. Or maybe I'm missing something

Good point. I think we are looking at an Ops experiment that probably is not yet on that version of things. We may want this "store the binary" for that use case in the mean time. It simplifies script logic as everything else is a symlink, but for these we run an executable to convert to binary at run time.

[gmao-rreichle]

We could pretty easily retrofit any older bcs with the nc4 version of the tile file. I really don't think we want a binary file floating around that's difficult to read/interpret.
The problem with abandoning txt is that the results aren't zero-diff -- if the baseline uses the txt file, using the nc4 file will result in a comparison failure (same for txt and binary). This may be an issue for testing and development more than application. When we introduced the nc4 file, we made it the default for new versions of bcs (at least in GEOSldas; not sure about GEOSgcm, and the DAS is in its own world anyway)

[mathomp4]

I am currently running a sweep of the tile files and converting to binary. I had to make a minor change to binarytile.F90 here:

GEOS-ESM/GEOS_Util#192

to make it a bit more flexible.

[mathomp4]

Also, I did a few tests and it looks like ifort, ifx, and GNU all make identical binary files for these. Which is nice.

[tclune]

We could pretty easily retrofit any older bcs with the nc4 version of the tile file. I really don't think we want a binary file floating around that's difficult to read/interpret.

@gmao-rreichle The intent here is to only use the binary file with those GEOS configurations that still use the binary file. The changes here will not affect anything that is expecting to find a netcdf file. It would be nice if we could just abandon the older tags that use .txt (and then make the .bin at run time), but we cannot. I can stop by if this clarification is insufficient.

[gmao-rreichle]

@tclune : I'm confused by the following:

The intent here is to only use the binary file with those GEOS configurations that still use the binary file.

Why "still" use the binary file? I'm not aware that we had binary tile files. I might be missing something. If we already have binary tile files floating around, so be it. But if this is a new thing, it makes absolutely no sense to me. We can talk this afternoon (after the 3pm mtg)

[mathomp4]

@tclune : I'm confused by the following:

The intent here is to only use the binary file with those GEOS configurations that still use the binary file.

Why "still" use the binary file? I'm not aware that we had binary tile files. I might be missing something. If we already have binary tile files floating around, so be it. But if this is a new thing, it makes absolutely no sense to me. We can talk this afternoon (after the 3pm mtg)

I'll try and answer a bit of the background here for when you talk with @tclune

The issue was brought up in @sshakoor1 's work in cleaning up how the GCM and ADAS link to BCs (see GEOS-ESM/GEOSgcm_App#823)

What we found is that in the GCM we do this:

1. Run linkbcs
2. Run binarytile.x to convert tile.data to tile.bin
3. Model reads tile.bin

In the ADAS this is the process:

1. Run gen_lnbcs.pl which makes lnbcs
2. Run lnbcs which internally runs binarytile.x
3. Model reads tile.bin

No matter what, we read the binary version.

So the question was: Should the new linkbcs.py script call binarytile.x internally (via subprocess) like lnbcs or should that step remain "outside" as is done in gcm_run.j?

Our thought was then, well, why even need to do that step at run time? We could precompute the binary versions of the text tile files once and then just link that in. At low-res, this is not much of a savings, but at very high res, it can be expensive (see above).

Then in the update from @sshakoor1 he can just do a simple "link" of the binary.

---

NOTE: If you look around the model, you'll see code like:

/bin/ln -sf $BCSDIR/geometry/$BCRSLV/${BCRSLV}-Pfafstetter.til  tile.data
if(     -e  $BCSDIR/geometry/$BCRSLV/${BCRSLV}-Pfafstetter.TIL) then
/bin/ln -sf $BCSDIR/geometry/$BCRSLV/${BCRSLV}-Pfafstetter.TIL  tile.bin
endif
...
./linkbcs

if (! -e tile.bin) then
$GEOSBIN/binarytile.x tile.data tile.bin
endif

At some point, someone had a convention that .til is text and .TIL is binary. This is Not A Good Idea™ especially on macOS which by default has a case insensitive filesystem.

So in my current "make binary" sweep, I'm making files named like:

CF2880x6C_CF2880x6C-Pfafstetter.til.bin

[gmao-rreichle]

Thanks, @mathomp4, for the additional insights. @tclune also explained things to me when we talked this afternoon. The upshot is that I was completely unaware of the txt-to-bin conversion tool as part of the GCM and ADAS setup. It definitely makes sense to synchronize the setup across the GCM and ADAS. Looking ahead, though, what we really need is for the model to read the tile info from the nc4 file. There's even a chance that the bits in the nc4 file are identical to those in the binary file. If that's the case, it'd be a zero-diff change.

[tclune]

There's even a chance that the bits in the nc4 file are identical to those in the binary file. If that's the case, it'd be a zero-diff change.

To expand on this. The diffs between .txt and .bin (and .nc4) are likely entirely due to conversion from string to binary when reading the text. But ... it must be verified of course.