Use HF safetensors by default

[leondz]

Support use_safetensors in hugging face models & enable by default

Verification

Little tricky since HF doesn't always go for pickles even if they're present and use_safetensors is disabled:

>>> g_s = pipeline("text-generation", model="gpt2", use_safetensors=False)
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 665/665 [00:00<00:00, 3.54MB/s]
model.safetensors:  67%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                                      | 367M/548M [00:31<00:16, 11.2MB/s]

[jmartin-tech]

Should this be expanded to set hf_args values? In the huggingface generator classes and in HFDetector?

Based on the current testing results and the finding noted about Callable signatures currently used I suspect this will need more refining to determine a valid approach since it seems unclear what models will have safe tensors.

[jmartin-tech]

A default not supplied by DEFAULT_PARAMS['hf_args'] should only be injected if the hf_constucutor is expected to support or pass thru the option:

The following methods are currently passed as hf_constructor:

transformers.pipeline
transformers.AutoConfig.from_pretrained
transformers.AutoModelForSeq2SeqLM.from_pretrained
transformers.LlavaNextForConditionalGeneration.from_pretrained

Unfortunately all the above methods mask use_safetensors as part of an optional **kwarg param.

This gathering pattern may not work for this option.

[leondz]

Yeah, that's exactly where I ended up with this also. Inspection doesn't immediately reveal the parameter, and support it not universal. There are some lists of what supports this but I prefer something that responds to the code at hand. More research needed.

[jmartin-tech]

Errant debugging statement?

[leondz]

extremely

[leondz]

see #990 (comment) - pausing for now