Lists of namelists cannot be added

[rhfogh]

A namelist containing a list of namelists created in Python cannot be written to file and throws an error.

The test program:

import f90nml
data = f90nml.Namelist()
data["dict1"] = f90nml.Namelist(attr1=1, attr2=2)
data["list1"] = []
data["list1"].append(f90nml.Namelist(attr3=3, attr4=4))
data["list1"].append(f90nml.Namelist(attr3=5, attr4=6))
data.write("somefile.out")

gives - for Python 3.8 and f90nml versions 1.4.1, 1.4.3, and 1.4.4

Traceback (most recent call last):
File "/home/rhfogh/Software/scripts/test_f90nml.py", line 9, in
data.write("/home/rhfogh/calc/mxcube_data/f90nmltst5.out")
File "/alt/rhfogh/Software/miniconda3/envs/mxcubeqt/lib/python3.8/site-packages/f90nml/namelist.py", line 578, in write
self._writestream(nml_file, sort)
File "/alt/rhfogh/Software/miniconda3/envs/mxcubeqt/lib/python3.8/site-packages/f90nml/namelist.py", line 677, in _writestream
self._write_nmlgrp(grp_name, grp_vars, nml_file, sort)
File "/alt/rhfogh/Software/miniconda3/envs/mxcubeqt/lib/python3.8/site-packages/f90nml/namelist.py", line 693, in _write_nmlgrp
for v_name, v_val in grp_vars.items():
AttributeError: 'list' object has no attribute 'items'

With the output:

&dict1
attr1 = 1
attr2 = 2
/

&list1

When running with Python 3.8 and f90nml version 1.3.1 the test works and you get the desired result:

&dict1
attr1 = 1
attr2 = 2
/

&list1
attr3 = 3
attr4 = 4
/

&list1
attr3 = 5
attr4 = 6
/

[marshallward]

I think that this line is problematic: data['list1'] = [].

f90nml 1.4 did a lot of work to improve support for namelist groups with the same name, but it also forced the interface to become more complicated. Repeated namelist groups now must be declared as "Cogroups" rather than treating them as lists.

This would be the new way to set up your namelist:

import f90nml

data = f90nml.Namelist()

data['dict1'] = f90nml.Namelist(attr1=1, attr2=2)

data.create_cogroup('list1')  # This is optional

data.add_cogroup('list1', f90nml.Namelist(attr3=3, attr4=4))
data.add_cogroup('list1', f90nml.Namelist(attr5=5, attr6=6))

I did try to preserve the existing behavior when accessing the Namelist (e.g. Namelist['list1'][0] should return the first one), but did not consider the construction of the Namelist with explicit lists. Maybe there is some way to restore this old behavior.

[marshallward]

Assuming there are no unexpected issues, I think we could modify things so that data['list1'] = [] is equivalent to data.create_cogroup('list1') and the data['list1'].append() calls are equivalent to data.add_cogroup('list1', ...).

[rhfogh]

@marshallward That would be great. Now that I understand how it ought to work (thanks!) I could also rewrite our code to use the cogroups, but it would be kind of neat that you can use the standard Python datatype, a list.

One question springs to mind, though: What should happen if you did data.create_cogroup('list1') followed by a single data.add_cogroup('list1', ...) call, or even none? Or, equivalently, if you added an empty list or a list with a single member? In normal Python logic you would expect the result to be an empty list, or a list with one member as appropriate, but with Fortran logic you might expect something else. If you cannot get the Python-expected behaviour from adding Python lists without confusing the Fortran expectations, it might be better to force people to use the cogroup syntax after all; that way there would be fewer surprises. If that is the preferred solution I would vote for a better error message, though, something about adding an illegal type.

PS I do not really know enough about f90nml, but I am assuming that when reading data back, data['list1'] would return a Python list - or something that behaved like it. If not, the ball game would change .

[marshallward]

Cogroup was originally meant to be an internal type that was invisible to the user, but I did eventually decide to create a small API during testing. It is a concise way to set them up, but it does add more concepts for the user to absorb.

data.create_cogroup('list1') sets up the metadata to track list1 as a cogroup. If there are no list1 groups, then data['list1'] returns an empty list. But if you iterate over values of data, then data will appear empty, with no reference to list1.

If list1 is declared to be a cogroup, then data['list1'] returns a list of Namelist values. If there's one list1 group, then it's a list of one. I can't recall if this is different from v1.3. Iteration will return the group, not the list.

In Fortran the namelist would be read serially, so data['list1'] would probably simply return the first group in the list, with subsequent reads giving the next in the list, and so forth, which I don't think anyone wants. I suppose we are trying to build a datatype that exists inbetween the Fortran and Python environments.

I think there is certainly a lot of nuance here, which you are doing a great job of exposing. My goal at the time was to support while preserving the old API for reading. It might be worth writing up a small document about Cogroups at some point.

Thanks very much for your feedback! I confess that I don't think much about namelist generation within Python. Unsurprisingly, I think most of the bug reports are from users who do this!

I would like to keep this issue open for a bit longer, while I test out list assignment to Namelist. If it doesn't affect the other uses of Namelist (or if I finally decide to split it into Namelist, Group, DerivedType), then I think it would work nicely.

[m-n-o]

Hello!

I have similar problem, trying to write multiple groups in namelist with the same name.

1. First try - As I understand it, this method will not work case the python dict definition.

nml_cfg_edit = dict(
        namrun=dict(
            nn_date0=start_date,
            cn_ocerst_indir=f"/path/{start_date}/",
        ),
        namsbc_blk=dict(
            cn_dir=f"/path/{start_date}/"
        ),
        nambdy_dta=
            dict(
                cn_dir=f"/path/{start_date}/"
            ),
        nambdy_dta=
            dict(
                cn_dir=f"/path/{start_date}/"
            ),
        nambdy_dta=
            dict(
                cn_dir=f"/path/{start_date}/"
            )
)

2. Second try - Doesn't work cause AttributeError: 'list' object has no attribute 'items' from f90nml/parser.py, line 416

nml_cfg_edit = dict(
        namrun=dict(
            nn_date0=start_date,
            cn_ocerst_indir=f"/path/{start_date}/",
        ),
        namsbc_blk=dict(
            cn_dir=f"/path/{start_date}/"
        ),
        nambdy_dta=[
            dict(
                cn_dir=f"/path/{start_date}/"
            ),
            dict(
                cn_dir=f"/path/{start_date}/"
            ),
            dict(
                cn_dir=f"/path/{start_date}/"
            )
        ]
)

Environment (conda):
python 3.9.19
f90nml 1.4.4

Is there a possibility at the moment to write multiple identical groups to namelist?

[marshallward]

@m-n-o For now, the only way to generate a Namelist with duplicate keys is to follow the Cogroup syntax, as in the example above.

Perhaps something like this:

from f90nml import Namelist
nml_cfg_edit = Namelist(namrun=Namelist(x=1,y=2), namsbc_blk=Namelist(u=1,v=2))
nml_cfg_edit.create_cogroup('nambdy_dta')
nml_cfg_edit.add_cogroup('nambdy_dta', Namelist(cn_dir=1))
nml_cfg_edit.add_cogroup('nambdy_dta', Namelist(cn_dir=2))
nml_cfg_edit.add_cogroup('nambdy_dta', Namelist(cn_dir=3))
print(nml_cfg_edit)

I agree that the syntax is a little odd. As mentioned above, it may be possible to improve the API and associate these functions with native list and dict dunder functions.

[m-n-o]

Thanks!
Your method is working and the Namelist object is being created. But I also have trouble with patching the original namelist file.
I can just read the source text, change the parameters and write it down, but this approach breaks the original formatting and comments. Therefore, I prefer patching.

Next code

f90nml.patch("/path/namelist_cfg", nml_cfg_edit, out_path="/path/namelist_cfg_tmplt")

causes next error: AttributeError: 'Cogroup' object has no attribute 'items'

[marshallward]

I suspect this is simply not possible at the moment. It would require modifying patch to work with Cogroups, which seems to not be possible at the moment.

I don't know if there will be time to look into this, but I'll open an issue on this topic.