ENH introduce SKRUB_DATA_DIRECTORY envar to control the data directory

[thomass-dev]

It should be awesome to control the data cache directory from another place than the python code itself.
Especially in a CI context, where we should ensure that the data is at the right place.

I propose to use envar to add another way to define the data directory.

[Vincent-Maladiere]

Hey @thomass-dev, thank you for the suggestion. Although this is not a big change, I'm not 100% sure we need this, as we use tempfile.TemporaryDirectory() in our own CI. Could you motivate this a little bit? Is this something you need for a project, e.g., skore? I don't have a strong opinion on this. WDYT @jeromedockes?

[jeromedockes]

I think that's definitely useful even for individual users (actually I would have guessed we already had that). Some users will want to choose the location of the data directory, eg to put it in a location shared with other users, on some different storage, somewhere that is not backed up, or simply to avoid cluttering their home directory. as this is a preference of the user / local machine, allowing to control it from the python code is not a solution, it should be a configuration file or env variable ( I prefer the env variable )

[Vincent-Maladiere]

as this is a preference of the user / local machine, allowing to control it from the python code is not a solution

I'm not sure I understand this bit, what do you mean?

[jeromedockes]

I mean the python code may be shared with other developers, whereas the directory is a user-specific preference. for example if I want to run the skrub examples but have them store the data in ~/whatever/datasets/skrub/ I cannot hardcode that location in the python scripts, I want to specify it through some other channel

[Vincent-Maladiere]

Ok, I understand your point. Just for the sake of argument, couldn't the shared python code also accept a directory parameter from the user and route that to skrub?

[jeromedockes]

yes you are right -- basically the options for passing that info to a program could be a command-line argument, an env variable, or a config file. as a user I want to set it once and forget about it, not pass it every time I invoke a script that uses skrub, and as a developer I don't want to add boiler plate to all my scripts to expose that argument (for example the skrub examples don't have it, and if they did sphinx wouldn't pass it when building the doc).

[Vincent-Maladiere]

Ok, that makes sense, thanks for detailing your thoughts.

[GaelVaroquaux]

Ok, that makes sense, thanks for detailing your thoughts.

I agree that the feature makes sense. Thanks!!

[thomass-dev]

I want to run the skrub examples but have them store the data in ~/whatever/datasets/skrub/ I cannot hardcode that location in the python scripts, I want to specify it through some other channel

This is exactly the purpose of this issue. Thanks @jeromedockes to have deep-dived in my mind 😄 .

To be explicit, my use-case is:

In skore we built example scripts using skrub that are intend to be shared to users. I want a fine control over data location, especially in the CI to make cache between pipelines (and on my personal computer), without hard-coding the location directly in the example scripts.

[Vincent-Maladiere]

Nice, could you share the link to this dev?