Support Makefile generation with src/configure.py

[technic960183]

Summary

This pull request adds support for automatically generating Makefiles for the analysis tools (gamer_compare_data, gamer_extract_profile, gamer_extract_uniform) alongside the main GAMER Makefile. It introduces a new command-line argument --tools for src/configure.py to enable this feature and refactors the tool Makefiles to use template variables, allowing them to be filled in with the correct build options during configuration.

Rationale

These changes make it easier to keep the analysis tools' build configurations in sync with the main codebase and simplify the process of configuring and building the tools for different environments. And this also helps the regression test module.

Changes

Tool Makefile generation

• Added a new --tools argument to the configuration script (src/configure.py) to optionally generate Makefiles for analysis tools.
• Implemented logic in src/configure.py to process a list of analysis tool paths, read their Makefile templates, replace template variables with actual configuration values, and write the finalized Makefiles.

Makefile template refactoring

• Renamed Makefile to Makefile_base for each analysis tool and replaced hardcoded build options and paths with template variables (e.g., @@@SIMU_OPTION@@@, @@@CXX@@@, @@@CXXFLAG@@@, @@@HDF5_PATH@@@, @@@MPI_PATH@@@). This allows these values to be set dynamically during configuration.

Usage

Run cd src && python3 configure.py --tools=true [other options] to use this feature.

Backward compatibility

• --tools is default to false and configure.py works the exactly the same as before when --tools is not given.
• Users can manually modified the Makefiles as they used to. Although those Makefiles are not tracked by git anymore.

Tests

• Regression test for GAMER to ensure backward compatibility
• Test on gamer_compare_data
• Test on gamer_extract_profile
• Test on gamer_extract_uniform

Discussion

• For the current design, --tools=true can be simplified as given --tools along. Or we make it be able to choice the tools to configure and maybe configure the tool only without alternating the Makefile of the main GAMER code. But is this necessary for now?
• The default value is false for --tools now. And we can make the default value configurable with feature from Add Default Machine Setting #383. This feature can be combined with the first point above to allow more flexible choice for users to their working scenario.

[hyschive]

@technic960183 I can see the advantages of this PR, but I also have a few concerns I’d like to raise.

• This PR breaks the independence among different tools and the main GAMER, which may introduce several potential issues and concerns. For examples:
  • Most of the compilation options in the main GAMER are actually irrelevant to tools.
  • What if we want to add tool-specific compilation options in the future?
  • In the main GAMER, configure.py sets NCOMP_PASSIVE_USER, which is then used to determine NCOMP_PASSIVE in Macro.h by including built-in passive scalars. However, in both gamer_extract_uniform and gamer_extract_profile, NCOMP_PASSIVE is set directly in the original Makefile. This inconsistency will cause failure when using the new Makefile generated by configure.py. Note that this is just one example of the kinds of problems that may arise when the tools and main GAMER follow different compilation logic.
• With this change, we also lose control over which compilation options are applied to each tool. While providing a generate_makefile.sh script for each tool might help, I’m sure if this is the best solution.

I'm not opposed this PR, and I'm open to any suggestions.

[technic960183]

@hyschive Thanks for the feedback. Here are my reply to the points you raised.

• Most of the compilation options in the main GAMER are actually irrelevant to tools.

I don't consider this as a problem for the users. Unless there are options (macros) which has different meaning between each tools and/or the main code.
Although I do agree that the code should be more explicated and precisely (i.e. exposing the relevant options only), it is limited by the structure of configure.py we having now. And I do not plan to do a major refactoring recently.

• What if we want to add tool-specific compilation options in the future?

For now, if those options added will not override anything in GAMER, it will just be fine for users.
Still, for developers, current design will decrease maintainability. The same statement applies as the previous point. To resolve this in a good way, we will need to modularize configure.py.

• In the main GAMER, configure.py sets NCOMP_PASSIVE_USER, which is then used to determine NCOMP_PASSIVE in Macro.h by including built-in passive scalars. However, in both gamer_extract_uniform and gamer_extract_profile, NCOMP_PASSIVE is set directly in the original Makefile. This inconsistency will cause failure when using the new Makefile generated by configure.py. Note that this is just one example of the kinds of problems that may arise when the tools and main GAMER follow different compilation logic.

I think the only way to resolve this is to find out all of these cases one by one, combining with the decision above, then the best design could be determined.

• With this change, we also lose control over which compilation options are applied to each tool. While providing a generate_makefile.sh script for each tool might help, I’m sure if this is the best solution.

Let me separate each tools and the main code. I would like to collect suggestions and usage patterns from the users who interact with these tools frequently in their daily work. This will help us determine the design of user interfaces.

E.g.
python configure.py --target=extract_profile [other options], python configure.py --target=all [other options]
or
python configure.py extract_profile [other options], python configure.py all [other options].

From the software-structure-designing point of view, I personally disfavor generate_make.sh, as it introduce a dependency on the path of python. Still, I think this can be discussed after resolving previous points.

Any comments, suggestions, and opinions are very welcome.