API: mode.nan_is_na to consistently distinguish NaN-vs-NA

[jbrockmendel]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

As discussed on the last dev call, this implements "mode.nan_is_na" (default True) to consider NaN as either always-equivalent or never-equivalent to NA.

This sits on top of

• TST: nan->NA in non-construction tests #62021, which trims the diff here by updating some tests to use NA instead of NaN.
• API: consistent NaN treatment for pyarrow dtypes #61732 which implements the option but only for pyarrow dtypes.
• API: improve dtype in df.where with EA other #62038 which addresses an issue in DataFrame.where
• BUG: read_csv with engine=pyarrow and numpy-nullable dtype #62053 which addresses a kludge in read_csv with engine="pyarrow"

Still need to

• Add docs for the new option, including whatsnew section
• deal with a kludge in algorithms.rank; fixed by API: rank with nullable dtypes preserve NA #62043
• deal with a kludge in read_csv with engine="pyarrow"; fixed by BUG: read_csv with engine=pyarrow and numpy-nullable dtype #62053
• Add tests for the issues this addresses

[jbrockmendel]

Discussed in the dev call before last where I, @mroeschke, and @Dr-Irv were +1. Joris was unenthused but "not necessarily opposed". On slack @rhshadrach expressed a +1. All those opinions were to the concept, not the execution.

[jbrockmendel]

gentle ping @mroeschke

[mroeschke]

Would we want the placeholder here to be nan for StringDtype(na_value=nan), i.e.

if ... and isinstance(dtype, ExtensionDtype):
    arrays = [dtype.na_value] * len(columns)

[jbrockmendel]

that'd probably be benign. would we expect pd.NA to ever not-work?

[mroeschke]

Yeah not sure if something like

df = pd.DataFrame({"a": ["b"]}, columns=["a", "b"], dtype=pd.StringDtype(na_value=np.nan))
df.loc[0, "b"]

Would correctly return nan here

[jbrockmendel]

yes it does

[jbrockmendel]

Updated to use the suggested idiom

[mroeschke]

Curious, I thought you were not fond of the environment variable pattern?

[jbrockmendel]

that does sound like the kind of opinion i would have, but ATM i don't find myself bothered by it

[jbrockmendel]

Gentle ping

[mroeschke]

Thanks @jbrockmendel