38863 Update Engineering Diffraction Calibration region of interest functionality

[andy-bridger]

Description of work

Summary of work

Added support so the Custom File provided to Engineering Diffraction Calibration can be an xml, such as those output by SaveDetectorsGrouping

Fixes #38863 .
Also fixes a bug I found in testing new implementation #38879

Further detail of work

Changed it so the interface now accepts xml files of the grouping. Additionally changed both the user facing and internal reference to this from CalFile (or variations upon this) to GroupingFile and in the UI Crop Calibration to Set Calibration Region of Interest, these provide more accurate names to actual functionality and should make the interface more intuitive to use.
Changed the naming suffix for custom file example_group.xml from _Custom to _Custom_example_group so they don't get overwritten when custom grouping is changed (also more clear to the user what grouping is being used).
Changed the naming suffix for custom crop example_spec from _Cropped to _Cropped_example_spec so they don't get overwritten when custom grouping is changed (also more clear to the user what grouping is being used).
Finally changed it for the vanadium normalisation so a warning is flagged if GROUP.CUSTOM or GROUP.CROPPED are loaded in from the ADS, which, together with more unique naming, should cover the bug #38879

To test:

Open the Engineering Diffraction Interface

Select Create New Calibration

Set the sample as 305738

Select Set Calibration Region of Interest

Set Region of Interest to Custom Grouping File in the drop down

Try setting a custom grouping in xml format (code to create one provided below)

Calibration should be successful

Try using calibration to focus on the next tab

If you use the customs below should get: 2 spectra for "bank", 10 for "module", and 90 for "block"

(Bank should give identical results to Test 1 in the manual testing: https://developer.mantidproject.org/Testing/EngineeringDiffraction/EngineeringDiffractionTestGuide.html)

from mantid.simpleapi import *

def get_detector_grouping_string(ws, group_by):
    info = ws.componentInfo()
    detinfo = ws.detectorInfo()
    dets = detinfo.detectorIDs()
    instr_dets = info.detectorsInSubtree(info.root())

    def get_det_id(comp_ind, dets, instr_dets):
        return dets[np.where(instr_dets == comp_ind)][0]

    nbi = info.indexOfAny("NorthBank")
    sbi = info.indexOfAny("SouthBank")


    nbmi = info.children(nbi)
    sbmi = info.children(sbi)

    nbmbi = [xx for x in [info.children(int(nbm)) for nbm in nbmi] for xx in x]
    sbmbi = [xx for x in [info.children(int(sbm)) for sbm in sbmi] for xx in x]
    if group_by == "bank":
        n_dets = "+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(nbi)])
        s_dets = "+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(sbi)])
        print([d for d in n_dets.split("+") if det_is_monitor(d, detinfo)])
        print([d for d in s_dets.split("+") if det_is_monitor(d, detinfo)])
        return ",".join([n_dets, s_dets])
    if group_by == "module":
        n_mods = ",".join(
            ["+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(int(nbm))]) for nbm in
             nbmi])
        s_mods = ",".join(
            ["+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(int(sbm))]) for sbm in
             sbmi])
        return ",".join([n_mods, s_mods])
    if group_by == "block":
        n_blocks = ",".join(
            ["+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(int(nbm))]) for nbm in
             nbmbi])
        s_blocks = ",".join(
            ["+".join([str(get_det_id(x, dets, instr_dets)) for x in info.detectorsInSubtree(int(sbm))]) for sbm in
             sbmbi])
        return ",".join([n_blocks, s_blocks])
        
ws = LoadNexus(r"C:\Users\kcd17618\Downloads\ENGINX00218189.nxs") #put any enginx run here

group_type = "module" #bank/module/block should all make different custom groupings

CreateGroupingWorkspace(ws,OutputWorkspace ="group_ws", 
                        CustomGroupingString=get_detector_grouping_string(ws, "module"), 
                        ComponentName = "ENGIN-X")
                        
SaveDetectorsGrouping("group_ws", f"custom_{group_type}.xml")

You can also run through the reproduction instructions here: #38879 and check that the bug is no longer there.

---

Reviewer

Please comment on the points listed below (full description).
Your comments will be used as part of the gatekeeper process, so please comment clearly on what you have checked during your review. If changes are made to the PR during the review process then your final comment will be the most important for gatekeepers. In this comment you should make it clear why any earlier review is still valid, or confirm that all requested changes have been addressed.

Code Review

• Is the code of an acceptable quality?
• Does the code conform to the coding standards?
• Are the unit tests small and test the class in isolation?
• If there is GUI work does it follow the GUI standards?
• If there are changes in the release notes then do they describe the changes appropriately?
• Do the release notes conform to the release notes guide?

Functional Tests

• Do changes function as described? Add comments below that describe the tests performed?
• Do the changes handle unexpected situations, e.g. bad input?
• Has the relevant (user and developer) documentation been added/updated?

Does everything look good? Mark the review as Approve. A member of @mantidproject/gatekeepers will take care of it.

Gatekeeper

If you need to request changes to a PR then please add a comment and set the review status to "Request changes". This will stop the PR from showing up in the list for other gatekeepers.

[JackEAllen]

I have not functionally tested yet, but from an initial pass through the proposed changes, everything looks good! - only a few minor suggestions.
I wonder if it would be worthwhile updating dev-docs/source/Testing/EngineeringDiffraction/EngineeringDiffractionTestGuide.rst to add a test to validate that #38879 and #38863 are not accidentally reintroduced in the future. I feel like this would be a good smoke test.

[JackEAllen]

While we are touching this section of code anyway, I think the logic here could be extracted into a separate method called something like _get_suffix to reduce duplication, maybe into something like this:

Suggested change

	# getters
	# getters
	def get_foc_ws_suffix(self) -> str:
	return self._get_suffix(GROUP_FOC_WS_SUFFIX)
	def get_group_suffix(self) -> str:
	return self._get_suffix(GROUP_SUFFIX)
	def get_group_ws_name(self) -> str:
	return self._get_suffix(GROUP_WS_NAMES)
	def _get_suffix(self, suffix_dict: dict) -> str:
	"""Get the suffix for the current group"""
	if self.group:
	if self.group != GROUP.CUSTOM:
	return suffix_dict[self.group]
	else:
	self.set_extra_group_suffix()
	return f"{suffix_dict[GROUP.CUSTOM]}{self.extra_group_suffix}"
	return ""

I can't comment of whether or not it makes sense to return "" if not self.group, as It may be better to return an warning instead.

[JackEAllen]

I think this may read nicer as: if self.prm_filepath:

[JackEAllen]

NIPICK: Because self.grouping_filepath can be set to None which .split would not be a valid attribute of, we should probably first assert that self.grouping_filepath is not None.
I would also be tempted to extract this into a separate method to reduce the number of if statements in this method and add some extra validation checks - maybe something like:

    def get_group_ws(self):
        """
        Returns grouping workspace for ROI (creates if not present)
        :return: group workspace
        """
        if not self.group_ws or not ADS.doesExist(self.group_ws.name()):
            if self.group.banks:
                self.create_bank_grouping_workspace()
            elif self.group == GROUP.CROPPED:
                self.create_grouping_workspace_from_spectra_list()
            elif self.group == GROUP.CUSTOM:
                self.create_grouping_workspace_from_file()
        return self.group_ws

    def create_grouping_workspace_from_file(self) -> None:
        """
        Create grouping workspace from a custom file (.cal or .xml)
        """
        if not self.grouping_filepath:
            raise ValueError("Grouping file path is not set.")
        ext = path.splitext(self.grouping_filepath)[-1].lower()
        if ext == ".cal":
            self.create_grouping_workspace_from_calfile()
        elif ext == ".xml":
            self.group_ws = LoadDetectorsGroupingFile(InputFile=self.grouping_filepath, OutputWorkspace=self.get_group_ws_name())
        else:
            raise ValueError(f"Unsupported file extension: {ext}")

[sf1919]

Genuine docs failure

[andy-bridger]

A bit of a further update, especially re: manual testing.

I was reluctant to add a manual test as it felt like it could be automatically tested and I know people don't love doing the manual testing. I have added a system test that is essentially carrying out the procedure of running a calibration and focus with one custom group file, then running it again with a different group - and checking that this results in a different final spec. This, combined with the new tests in test_EnggUtils should cover the bug from reappearing.

Re doc failure: I tried to change so the release notes link always has the algorithm name, but forgot to remove the <>

[JackEAllen]

Thanks for the changes and additional tests! A system test is a good idea to save on adding another smoke test.

Would you be happy to rebase you're commits to squash them down? From looking through them, I think we can reduce them and possible re-order here and there. Here's a nice blog by GitHub I have used as a reference for structuring my own commits in the past: https://github.blog/developer-skills/github/write-better-commits-build-better-projects/
I've added a few minor suggestions to condense some methods slightly. but overall, your changes look really good and work as expected from manual testing.

[JackEAllen]

Sorry for not picking this one up in the initial review. While we are here, I think we can actually simplify on_combo_changed quite a lot to condense it down so we don't have so many elif statements to hopefully make the method easier to read at a glance by using a dictionary to map groups and passing boolean comparisons of the indexes to configure custom_calfile_enabled, custom_spectra_enabled and set_custom_widgets_visibility

   def on_combo_changed(self, index: int) -> None:
        """
        Handles the event when the combo box selection is changed.
        GROUP.TEXTURE30 is the default group if no cropping is requested.

        :param index: The index of the selected item in the combo box.
        """
        group_mapping = {
            0: GROUP.CUSTOM,
            1: GROUP.NORTH,
            2: GROUP.SOUTH,
            3: GROUP.CROPPED,
            4: GROUP.TEXTURE20,
            5: GROUP.TEXTURE30
        }
        self.group = group_mapping.get(index, GROUP.TEXTURE30)
        self.custom_groupingfile_enabled = (index == 0)
        self.custom_spectra_enabled = (index == 3)
        self.set_custom_widgets_visibility(index == 0, index == 3)

As this refactor suggestion is technically out of scope for the assigned issue, feel free to ignore. This is more of a personal preference thing for trying to clean up code as I'm modifying it so a codebase can slowly be cleaned up/kept up to date over time.

[JackEAllen]

NITPICK: I would be tempted to re-order the logic here a little and to add a description of the type error should this get triggered - maybe to something similar to:

    def set_extra_group_suffix(self) -> None:
        self.extra_group_suffix = ""
        if not isinstance(self.grouping_filepath, str):
            raise TypeError("The grouping file path must be a string")
        if self.group == GROUP.CUSTOM:
            self.set_grouping_filepath_from_prm_filepath()
            filename = path.basename(self.grouping_filepath)
            self.extra_group_suffix = f"_{path.splitext(filename)[0]}"
        elif self.group == GROUP.CROPPED and self.spectra_list_str:
            self.extra_group_suffix = f"_{self.spectra_list_str}"

What are your thoughts on this?

[JackEAllen]

can we move the comments into the docstring