User supplied @-variables

[nizomovs]

It would be nice to add support for user-defined @-variables. They could be provided in the experiments.yml or as a separate file. simex would then use those custom @-variables and replace them similarly to the simex @-variables.

If the custom @-variables were to be defined in the experiments.yml, it could look like this:

# The 'variables' field is a key-value-map of the custom @-variables
variables:
  - KEY1: value1
  - KEY2: value2

experiments:
  - name: test
    args: ['echo', '@KEY1@', '@EXTRA_ARGS@']
    exclusive: true
    num_nodes: 8
    stdout: out

variants:
  - axis: 'keys'
    items:
      - name: 'key1'
        extra_args: ['@KEY1@']
      - name: 'key2'
        extra_args: ['@KEY2@']

Running these experiments should output value1 value1 and value2 value respectively.

This could also solve #174: One could have a custom @-variable RUNTIME: srun and then use it in the experiment setup args: ['@RUNTIME@', ..., '@EXTRA_ARGS@']

[bernlu]

First, I do not understand how the example produces the described output. To me, this is what I expect to happen:
There are two runs, one each for the two items in axis keys. For key1, the call would then be echo @key1@ @key1@ = echo value1 value1, and for key2 echo @key1@ @key2@ = echo value1 value2.

Second, there is already a feature in YAML to define and use variables: Node Anchors (https://yaml.org/spec/1.2.2/#692-node-anchors). I think it would be better to use these features instead of defining our own. Your example with the node anchor syntax:

variants:
  - axis: 'keys'
    items:
      - name: 'key1'
        extra_args: [&KEY1 'value1']
      - name: 'key2'
        extra_args: [&KEY2 'value2']

experiments:
  - name: test
    args: ['echo', *KEY1, '@EXTRA_ARGS@']
    exclusive: true
    num_nodes: 8
    stdout: out