mail-type args not being set correctly on Perlmutter

[whannah1]

Recently I've noticed that I don't get any emails from my E3SM jobs on perlmutter when they finish successfully, but, oddly, I haven't heard of any other E3SM users having this problem. I went looking for an explanation and noticed that my jobs are being submitted with two separate mail-type arguments (--mail-type end --mail-type fail) instead of a single argument with comma separated values (--mail-type end,fail).

Looking through the CIME source it took awhile to find where these arguments are actually constructed in cime/CIME/XML/env_batch.py. Below is the specific block that sets the --mail-type  flag:

        if mail_type:
            mail_type_flag = self.get_value("batch_mail_type_flag", subgroup=None)
            if mail_type_flag is not None:
                mail_type_args = []
                for indv_type in mail_type:
                    mail_type_arg = self.get_batch_mail_type(indv_type)
                    mail_type_args.append(mail_type_arg)

                if mail_type_flag == "-m":
                    # hacky, PBS-type systems pass multiple mail-types differently
                    submitargs += " {} {}".format(
                        mail_type_flag, "".join(mail_type_args)
                    )
                else:
                    submitargs += " {} {}".format(
                        mail_type_flag,
                        " {} ".format(mail_type_flag).join(mail_type_args),
                    )

The line in the else block is the key one - and by changing it to this:

                    submitargs += " {} {}".format(
                        mail_type_flag,
                        ",".join(mail_type_args),
                    )

I see that the argument changes from two separate args to comma separated values:
i.e. --mail-type end --mail-type fail  >>> --mail-type end,fail

So I guess this leaves me with a few questions:

• is this a reasonable change to make?
• how has this not affected anyone else?
• was there a slurm update that changed how it parses these arguments?

Any thoughts? @jedwards4b @rljacob @jasonb5 @billsacks @jgfouca

[jgfouca]

@whannah1 , yeah, this seems worth a look.