Model discussion - version 0.0.1

[rhfogh]

We should use discussions to evaluate and change model versions, as changes to the Wiki do not get notified.

There is a first draft of how we might want to do the modelling on the Wiki, version 0.0.1.
Please add your comments.

[rhfogh]

Clemens Vonrhein commented on the proposal:

A set of initial (random) comments:

What is the relation between DataClasses (DataClass?) and DataSets
(DataSet?)? I can see how a DataSetGroup is a group of DataSets - but
the naming "DataSet" seems a bit unfortnuate to me: it sounds to much
like dataset in the traditional/old sense. It might be better to use
something without any direct match to typical MX lingo ... maybe just
DataGroup as a group of Data (since Data is plural, i.e. containing
different items)?

Or maybe your DataSet, DataSetGroup and the ICAT DataCollection and
Dataset are meant to match with MX language? Of course, your camelcase
notation is a bit cleaner than the ICAT one and might already be
enough to clarify these are abstract data model concepts and not
necessarily MX experiment descriptors.

I like the concept of using same classes for input and output: that
avoids most confusion - but it isn't quite clear to me how one would
mark a value (or class?) as coming from an input versus an output
type? Or maybe something described as "input" must after all be coming
out of "something" so it actually is output in some sense already
anyway, right?

Regarding axis convention on the detector: it would be good to stick
with McStas (if plugging into the whole HDRMX, Nexus/NXmx system) -
but maybe also allow somewhere an overal transformation from the
coordinate system used to McSTAS. This way any site could use whatever
their coordinate system is (often different from McSTAS) while
ensuring that any system will always be defined in relation to a
standard ... and if they already use McSTAS that transformation would
just be the unit matrix.

Looking at the tables:

• what does datatype in UnitCell represent?

• it might be useful to provide _esd/_su items for any parameter
  that could also be the result of some experimental determination
  (e.g. unit cell parameters on output)?

• MXScan:

  • energy could benefit from an error estimate (that can be used in
    autoPROC to define when two wavlengths are "the same"

  • do we need flux_start and flux_end (if that can be provided)?

• ResolutionBin

  • I'm not too happy with the name ... but not sure about a better
    one yet

  • Can we maybe drop r_merge at this point? I'd love that!!

  • *_ellipsoidal only make sense with a relation to the ellipsoid
    definition (which will probably be in some other table)?

  • should specify if Friedel pairs were kept separate or not here
    (friedels_law: boolean)?

• MX Processing

  • resolution_cutoff*: better naming could be data_cutoff_* or such
    (to avoid the historical baggage of "resolution")?

  • should probably contain the full ellipsoid definition here (see
    above)

[rhfogh]

Re DataSet:

DataSet is the ICAT name, and basically describes a file or files with attached parameters.
DataSetGroup corresponds to the ICAT DataCollection class (renamed, as that name was just too confusing). Better names might be a good idea.

Re datatype:

The datatype of UnitCell is just 'UnitCell'. The the idea is that you must be able to query which kind of object you have - if you can do that without having this particular attribute we can skip it. This is not that needed for UnitCell, but it would be needed to distinguish between different (sub)types of DataSet, e.g. MXScan, and MeshScan.

[KarlLevik]

Thank you, Rasmus, for your efforts!

If I'm reading model 0.0.1 correctly, it doesn't seem to support multiple Samples for one MXProcessing. Is that something which might be needed?

Re: a datatype attribute in the various classes, I suspect this won't be needed, as you can surely get this from parsing the JSON or XML or whatever markup language is used.

I'm a bit confused about the spreadsheet which has some attributes with datatype str, yet their "unit" is enum. Shouldn't the datatype be enum? And in the diagram, DataSetGroup.role is shown as "str" whereas MXProcessing.binning_mode is "enum". However, in the spreadsheet, they both have type "str" and unit "enum".

Maybe another name for data set / data collection could be Measurement. I'm just throwing it out there, just in case!

Although I can see the usefulness of it for the apps/users, I'm not immediately a fan of the SpecificData class which to me feels a bit like a bodge. This allows for sending random attributes which can only be stored in some kind of dictionary or key-value store/table on the back-end side. We might as well not bother specifying attributes in the model because it can just be thrown into SpecificData.

Finally, I note that the class names are PascalCase whereas the attributes are snake_case. Is that a deliberate decision? Won't snake_case look a bit awkward in JSON?

I have some other thoughts as well, but I'll keep them to myself for now!

[rhfogh]

The Sample is a good point. The ICAT data model has the sample linked to the DataSet. For the simplest case (one sample per experiment or processing) it makes more sense to have it on the DataSetGroup, but I was aware that this might not hold. I do not know the relevant use cases well enough, but if someone can list a couple of those, I can try to make an improved model here.

str and enum is me not being on top of the proper vocabulary. What you would want in a program amounts to one of a closed list of strings. Should I just say 'enum' in that case?

DataSet/DataSetGroup are indeed not very good names. One problem with 'Measurement' as a replacement is that the same class or its subtypes would also be used for processing output. Let us see what we can find to improve this.

The CamelCase v. snake_case was just me continuing my Python habits. Let us talk about what casing we should use.

I think SpecificData are invaluable as an extension mechanism. For instance Global Phasing has been arguing for years about being able to store anisotropic diffraction limits and other things that other programs do not produce, without ever getting anywhere. With SpecificData at least we would be able to store them, and eventually they could be promoted to the full model. In an ideal world any new attributes or changes would immediately be added to the model proper, with full agreement by everyone, so that there was no need for these (admittedly hacky) mechanisms. We should still aim for that, but in the real world an extension mechanism is at least better than unilateral model changes, misuse of existing attributes, or paralysis.

[KarlLevik]

I think it could make sense to refer/link to the sample from MXExperiment, but not from MXProcessing.

So, a processing program could figure out which samples are used by looking at the MXScans in the MXProcessing, and then find the sample by looking at the MXExperiment parents of those MXScans.

Alternatively, or in addition, the MXScans could directly refer to a Sample.

[rhfogh]

I have added some overview information on relevant ICAT classes for reeference.