config file

[leondz]

different clauses per class

provides defaults for all plugins

provide one config file, allow for another config file to be created that can override these

e.g. garak.defaults.conf and also garak.conf

add a .gitignore for the local, non-global one

hide the global one in a subdir to discourage editing, add a notice that it'll get clobbered

[leondz]

What do we want to configure?

• Keep overall params distinct from run-specific params.
  • What are overall params? Things that, if changed, will still give the same reported security. Things like report save location, verbosity level, default # generations - things that don't affect
  • What are run-specific params? Params that frame/describe a single run overall, e.g. generator, harness, probes, maybe detectors, buffs, seed
    the content of a run
• Consider setting plugin defaults within the plugins themselves, and then a site config and run config can override them
• How do people know what settings are possible within each plugin?
  • Perhaps defined/expressed in plugin_info - configurable params need to be distinct from non-configurable
  • Maybe a mutability thing, or decorator, or there's a whitelist of core params, or difference between class attribs and vars, or..

Sounds like there'll be a hierarchy, where:

plugin code < base config < site config < run config < cli params

• run configs are 100% user created and passed through CLI / garak console
• site configs are editable, in .gitignore and there's a default location examined
• base configs are not editable. stash them in the dir structure, add a comment at the top
• we're not going to look for API keys in these files - env vars are fine/better

dynaconf already supports overrides (https://www.dynaconf.com/faq/#default-override-workflow), could work fine

how do we define what's configurable in a plugin? maybe label non-configurable values as immutable / internal - https://realpython.com/python-mutable-vs-immutable-types/#techniques-to-control-mutability-in-custom-classes

[leondz]

CLI flow:

• load _config.settings (base+site?)
• use _config.settings to assert defaults in CLI
• take values from CLI options to _config.settings
• delete args (maybe have it point to _config.settings?)

[leondz]

So far so good. We also need to consider setting per-plugin vars.

• How can these be exposed & populated?
• What kind of syntax conventions are there already? E.g. max_len is around as a pretty popular global param, and should probably be in the core config
• Definitely worth splitting up plugins section into generators, probes, detectors, buffs

[leondz]

pr #311, #312