Mpi omp support (#343)

* #3 Introduce fixtures for Fortran compiler and tool_box.

* #3 Remove explicit compiler information from MpCommonArgs (since it's already part of the config toolbox).

* #3 Added linker as tool.

* #3 Added test for linking shared libraries.

* #3 Pass compiler flags to the linker if a compiler was specified.

* #3 Remove unused function.

* #3 Removed more unused code.

* #3 Automatically add a linker for each compiler.

* #3 Fixed typo.

* #3 Support vendor for compiler and linker.

* #3 Make linker having a vendor, too.

* #3 Add set_default_vendor method to tool repository.

* Ignore build directory for git.

* Updated test.

* # Fix some mypy errors and warnings.

* Avoid using get() for singleton, instead use __new__ which makes mypy happier.

* Changed the transformation_script parameter of function psyclone to accept a function that can return file-specific transformation scripts

* Make mypy happy by using patch.object.

* Remove more comments and confusion about mypy :)

* Try to make mypy happy on older python versions.

* Make flake8 happy.

* Sort imported name alphabetically.

* Try to fix failing hash test (and add some additional improvements in the test).

* Removed fpath= for input transformation_script function to pass mypy test for Python 3.7; Moved transformation_script_hash test to unit test from system test

* Fix mypy typing check errors for psyclone unit test

* Fix config typing issue with mypy in psyclone unit test

* Fix flake8 issues; Revert Config mypy typing fix

* Add comment to ignore typing check for fpath parameter of input transformation_script function

* Fix assert check after transformation_script function is changed from being called twice to once

* Filter out 'no transformation script' warning for psyclone system test

* Replace 'ignore' typing of fpath of transformation_script with removing keyword argument

* #3 Support proper tests to check if tools are available.

* 1. Updated transformation_script description; 2. Modified mock_transformation_script; 3.Removed redundant _analysis_for_prebuilds

* Updated lfric/atm.py and lfric/gungho.py examples to pass in transformation_script functions

* Added description for the psyclone step to instructions on writing a config

* #3 Added git as a tool.

* #3 Fix incorrect | usage in typing.

* #3 Added unit tests for git.

* #3 Renamed git.py to versioning.py, to avoid name clash with the corresponding test_git.py tests.

* #3 Converted svn and fcm to tools.

* #3 Fixed missing whitespace.

* Modified the documentation for writing a config with PSyclone

* Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests;
Changed the logic of the transformation_script examples

* #3 Replaced ar with tool object.

* #3 Added tests for ar.py.

* #3 Removed debug output.

* #3 Converted PSyclone to be a tool.

* #3 Removed debug print, fixed python 3.7 typing information.

* #3 Updated comments.

* Modified the get_optimisation_script function examples and updated the doc formatting

* #3 Add Rsync tool.

* #3 Removed now unused function.

* #3 Added test for rsync.

* #3 Fixed all mypy warnings about functions not checked.

* #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well.

* #3 Remove duplicated flags.

* #3 Fixed changed order of linking.

* #3 Removed run_command function.

* #3 Fixed 3.8 typing error.

* #3 Fixed unused imports.

* #3 Move flags checksum into Flags, and remove now unused tools.py file.

* #3 Renamed newtools to tools.

* #3 Made custom function for all git functions called (instead of just calling run).

* #3 Updated and fixed comments.

* #3 Fixed errors in comments.

* Fixed minor errors in documentation.

* #3 Make it easier to create wrapper around standard compiler.

* #3 Added documentation for all tool related classes and their usage.

* #3 Added MISC category.

* Addressed reviewer's comments.

* Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option.

* Fixed mypy failures, including changes to import statement to avoid cyclic imports :(.

* #3 Fix circular import.

* Added #TODO so that this can be removed once fparser supports sentinels.

* Fix typing problems by ignoring fparser.

* Replaced more string names for artefacts with enums.

* Removed EXECUTABLES from constants.

* Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet.

* Moved OBJECT_FILES from constants into ArtefactSet.

* Moved OBJECT_ARCHIVES from constants to ArtefactSet.

* Moved PRAGMAD_C from constants to ArtefactSet.

* Turned 'all_source' into an enum.

* Allow integer as revision.

* Fixed flake8 error.

* Removed specific functions to add/get fortran source files etc.

* Removed non-existing and unneccessary collections.

* Try to fix all run_configs.

* Fixed rebase issues.

* Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file.

* Started to replace artefacts when files are pre-processed.

* Removed linker argument from linking step in all examples.

* Try to get jules to link.

* Fixed build_jules.

* Fixed other issues raised in reviews.

* Try to get jules to link.

* Fixed other issues raised in reviews.

* Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone.

* Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections.

* Fixed some examples.

* Fix flake8 error.

* Fixed failing tests.

* Support empty comments.

* Fix preprocessor to not unnecessary remove and add files that are already in the output directory.

* Allow find_soure_files to be called more than once by adding files (not replacing artefact).

* Updated lfric_common so that files created by configurator are written in build (not source).

* Use c_build_files instead of pragmad_c.

* Removed unnecessary str.

* Documented the new artefact set handling.

* Fixed typo.

* Make the PSyclone API configurable.

* Fixed formatting of documentation, properly used ArtefactSet names.

* Support .f and .F Fortran files.

* Removed setter for tool.is_available, which was only used for testing.

* #3 Fix documentation and coding style issues from review.

* Renamed Categories into Category.

* Minor coding style cleanup.

* Removed more unnecessary ().

* Re-added (invalid) grab_pre_build call.

* Fixed typo.

* Renamed set_default_vendor to set_default_compiler_suite.

* Renamed VendorTool to CompilerSuiteTool.

* Also accept a Path as exec_name specification for a tool.

* Move the check_available function into the base class.

* Fixed some types and documentation.

* Fix typing error.

* Added explanation for meta-compiler.

* Improved error handling and documentation.

* Replace mpiifort with mpifort to be a tiny bit more portable.

* Use classes to group tests for git/svn/fcm together.

* Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication.

* Code improvement as suggested by review.

* Fixed run config

* Added reference to ticket.

* Updated type information.

* More typing fixes.

* Fixed typing warnings.

* As requested by reviewer removed is_working_copy functionality.

* Issue a warning (which can be silenced) when a tool in a toolbox is replaced.

* Fixed flake8.

* Fixed flake8.

* Fixed failing test.

* Addressed issues raised in review.

* Removed now unnecessary operations.

* Updated some type information.

* Fixed all references to APIs to be consistent with PSyclone 2.5.

* Added api to the checksum computation.

* Fixed type information.

* Added test to verify that changing the api changes the checksum.

* Make compiler version a tuple of integers

* Update some tests to use tuple versions

* Explicitly test handling of bad version format

* Fix formatting

* Tidying up

* Make compiler raise an error for any invalid version string

Assume these compilers don't need to be hashed.
Saves dealing with empty tuples.

* Check compiler version string for compiler name

* Fix formatting

* Add compiler.get_version_string() method

Includes other cleanup from PR comments

* Add mpi and openmp settings to BuildConfig, made compiler MPI aware.

* Looks like the circular dependency has been fixed.

* Revert "Looks like the circular dependency has been fixed." ...
while it works with the tests, a real application still triggered it.

This reverts commit 150dc37.

* Don't even try to find a C compiler if no C files are to be compiled.

* Updated gitignore to ignore (recently renamed) documentation.

* Fixed failing test.

* Return from compile Fortran early if there are no files to compiles. Fixed coding style.

* Add MPI enables wrapper for intel and gnu compiler.

* Fixed test.

* Automatically add openmp flag to compiler and linker based on BuildConfig.

* Removed enforcement of keyword parameters, which is not supported in python 3.7.

* Fixed failing test.

* Support more than one tool of a given suite by sorting them.

* Use different version checkout for each compiler vendor with mixins

* Refactoring, remove unittest compiler class

* Fix some mypy errors

* Use 'Union' type hint to fix build checks

* Return run_version_command to base Compiler class

Provides default version command that can be overridden for other compilers.
Also fix some incorrect tests
Other tidying

* Add a missing type hint

* Remove inheritance from mixins and use protocol

* Simplify compiler inheritance

Mixins have static methods with unique names,
overrides only happen in concrete classes

* Simplify usage of compiler-specific parsing mixins.

* Test for missing mixin.

* Fixed test.

* Added more tests for invalid version numbers.

* Added more test cases for invalid version number, improved regex to work as expected.

* Fixed typo in test.

* Fixed test.

* Split tests into smaller individual ones, fixed missing asssert in test.

* Parameterised compiler version tests to also test wrapper.

* Added missing MPI parameter when getting the compiler.

* Fixed comments.

* Order parameters to be in same order for various compiler classes.

* Remove stray character

* Made mpi and openmp default to False in the BuildConfig constructor.

* Removed white space.

* Support compilers that do not support OpenMP.

* Added documentation for openmp parameter.

---------

Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: jasonjunweilyu <[email protected]>
Co-authored-by: Luke Hoffmann <[email protected]>
Co-authored-by: Luke Hoffmann <[email protected]>

.gitignore

@@ -4,9 +4,9 @@ __pycache__/
 *$py.class
 
 # Build directory for documentation
-docs/build
-docs/source/api
-docs/source/apidoc
+Documentation/build
+Documentation/source/api
+Documentation/source/apidoc
 
 # C extensions
 *.so

Documentation/source/site-specific-config.rst

@@ -148,6 +148,18 @@ rsync, ar, ...).
     tool_box = ToolBox()
     default_c_compiler = tool_box.get_tool(Category.C_COMPILER)
 
+There is special handling for compilers and linkers: the build
+configuration stores the information if an MPI and/or OpenMP build
+is requested. So when a default tool is requested by the ToolBox
+from the ToolRepository (i.e. when the user has not added specific
+compilers or linkers), this information is taken into account, and
+only a compiler that will fulfil the requirements is returned. For
+example, if you have `gfortran` and `mpif90-gfortran` defined in this
+order in the ToolRepository, and request the default compiler for an
+MPI build, the `mpif90-gfortran` instance is returned, not `gfortran`.
+On the other hand, if no MPI is requested, an MPI-enabled compiler
+might be returned, which does not affect the final result, since
+an MPI compiler just adds include- and library-paths.
 
 TODO
 ====

run_configs/gcom/build_gcom_ar.py

@@ -15,7 +15,7 @@
 if __name__ == '__main__':
 
     with BuildConfig(project_label='gcom object archive $compiler',
-                     tool_box=ToolBox()) as state:
+                     mpi=True, openmp=False, tool_box=ToolBox()) as state:
         common_build_steps(state)
         archive_objects(state, output_fpath='$output/libgcom.a')
         cleanup_prebuilds(state, all_unused=True)

run_configs/gcom/build_gcom_so.py

@@ -20,7 +20,7 @@
     parsed_args = arg_parser.parse_args()
 
     with BuildConfig(project_label='gcom shared library $compiler',
-                     tool_box=ToolBox()) as state:
+                     mpi=True, openmp=False, tool_box=ToolBox()) as state:
         common_build_steps(state, fpic=True)
-        link_shared_object(state, output_fpath='$output/libgcom.so'),
+        link_shared_object(state, output_fpath='$output/libgcom.so')
         cleanup_prebuilds(state, all_unused=True)

run_configs/jules/build_jules.py

@@ -42,12 +42,15 @@ def __init__(self):
     tool_box.add_tool(Linker(compiler=fc))
 
     with BuildConfig(project_label=f'jules {revision} $compiler',
-                     tool_box=tool_box) as state:
-        # grab the source. todo: use some checkouts instead of exports in these configs.
-        fcm_export(state, src='fcm:jules.xm_tr/src', revision=revision, dst_label='src')
-        fcm_export(state, src='fcm:jules.xm_tr/utils', revision=revision, dst_label='utils')
+                     mpi=False, openmp=False, tool_box=tool_box) as state:
+        # grab the source. todo: use some checkouts instead of exports
+        # in these configs.
+        fcm_export(state, src='fcm:jules.xm_tr/src', revision=revision,
+                   dst_label='src')
+        fcm_export(state, src='fcm:jules.xm_tr/utils', revision=revision,
+                   dst_label='utils')
 
-        grab_pre_build(state, path='/not/a/real/folder', allow_fail=True),
+        grab_pre_build(state, path='/not/a/real/folder', allow_fail=True)
 
         # find the source files
         find_source_files(state, path_filters=[
@@ -61,9 +64,11 @@ def __init__(self):
         # move inc files to the root for easy tool use
         root_inc_files(state)
 
-        preprocess_fortran(state, common_flags=['-P', '-DMPI_DUMMY', '-DNCDF_DUMMY', '-I$output'])
+        preprocess_fortran(state, common_flags=['-P', '-DMPI_DUMMY',
+                                                '-DNCDF_DUMMY', '-I$output'])
 
-        analyse(state, root_symbol='jules', unreferenced_deps=['imogen_update_carb'])
+        analyse(state, root_symbol='jules',
+                unreferenced_deps=['imogen_update_carb'])
 
         compile_fortran(state)
 

run_configs/lfric/atm.py

@@ -172,7 +172,7 @@ def file_filtering(config):
     gpl_utils_source = gpl_utils_source_config.source_root / 'gpl_utils'
 
     with BuildConfig(project_label='atm $compiler $two_stage',
-                     tool_box=ToolBox()) as state:
+                     mpi=False, openmp=False, tool_box=ToolBox()) as state:
 
         # todo: use different dst_labels because they all go into the same folder,
         #       making it hard to see what came from where?

run_configs/lfric/grab_lfric.py

@@ -16,10 +16,12 @@
 # these configs are interrogated by the build scripts
 # todo: doesn't need two separate configs, they use the same project workspace
 tool_box = ToolBox()
-lfric_source_config = BuildConfig(project_label=f'lfric source {LFRIC_REVISION}',
-                                  tool_box=tool_box)
-gpl_utils_source_config = BuildConfig(project_label=f'lfric source {LFRIC_REVISION}',
-                                      tool_box=tool_box)
+lfric_source_config = BuildConfig(
+    project_label=f'lfric source {LFRIC_REVISION}',
+    tool_box=tool_box)
+gpl_utils_source_config = BuildConfig(
+    project_label=f'lfric source {LFRIC_REVISION}',
+    tool_box=tool_box)
 
 
 if __name__ == '__main__':

run_configs/lfric/gungho.py

@@ -33,7 +33,7 @@
     gpl_utils_source = gpl_utils_source_config.source_root / 'gpl_utils'
 
     with BuildConfig(project_label='gungho $compiler $two_stage',
-                     tool_box=ToolBox()) as state:
+                     mpi=True, openmp=True, tool_box=ToolBox()) as state:
         grab_folder(state, src=lfric_source / 'infrastructure/source/', dst_label='')
         grab_folder(state, src=lfric_source / 'components/driver/source/', dst_label='')
         grab_folder(state, src=lfric_source / 'components' / 'inventory' / 'source', dst_label='')
@@ -87,7 +87,7 @@
             state,
             common_flags=[
                 '-c',
-                '-ffree-line-length-none', '-fopenmp',
+                '-ffree-line-length-none',
                 '-g',
                 '-std=f2008',
 
@@ -104,8 +104,6 @@
         link_exe(
             state,
             flags=[
-                '-fopenmp',
-
                 '-lyaxt', '-lyaxt_c', '-lnetcdff', '-lnetcdf', '-lhdf5',  # EXTERNAL_DYNAMIC_LIBRARIES
                 '-lxios',  # EXTERNAL_STATIC_LIBRARIES
                 '-lstdc++',

run_configs/lfric/mesh_tools.py

@@ -25,7 +25,7 @@
     psyclone_overrides = Path(__file__).parent / 'mesh_tools_overrides'
 
     with BuildConfig(project_label='mesh tools $compiler $two_stage',
-                     tool_box=ToolBox()) as state:
+                     mpi=True, openmp=False, tool_box=ToolBox()) as state:
         grab_folder(state, src=lfric_source / 'infrastructure/source/', dst_label='')
         grab_folder(state, src=lfric_source / 'mesh_tools/source/', dst_label='')
         grab_folder(state, src=lfric_source / 'components/science/source/', dst_label='')

run_configs/tiny_fortran/build_tiny_fortran.py

@@ -33,13 +33,13 @@ def __init__(self):
     with BuildConfig(project_label='tiny_fortran $compiler',
                      tool_box=tool_box) as state:
         git_checkout(state, src='https://github.com/metomi/fab-test-data.git',
-                     revision='main', dst_label='src'),
+                     revision='main', dst_label='src')
 
-        find_source_files(state),
+        find_source_files(state)
 
-        preprocess_fortran(state),
+        preprocess_fortran(state)
 
-        analyse(state, root_symbol='my_prog'),
+        analyse(state, root_symbol='my_prog')
 
-        compile_fortran(state),
-        link_exe(state),
+        compile_fortran(state)
+        link_exe(state)

run_configs/um/build_um.py

@@ -124,8 +124,11 @@ def replace_in_file(inpath, outpath, find, replace):
     revision = 'vn12.1'
     um_revision = revision.replace('vn', 'um')
 
-    state = BuildConfig(project_label=f'um atmos safe {revision} $compiler $two_stage',
-                        tool_box=ToolBox())
+    # The original build script disabled openmp, so for now
+    # we keep this disabled.
+    state = BuildConfig(
+        project_label=f'um atmos safe {revision} $compiler $two_stage',
+        mpi=True, openmp=False, tool_box=ToolBox())
 
     # compiler-specific flags
     compiler = state.tool_box[Category.FORTRAN_COMPILER]