Introduce a new links category for the YAML definitions

doc/datamodel_syntax.md

@@ -155,6 +155,38 @@ define which `Types` can be used with this interface class, in this case the
 not allow for mutable access to their data.** They can be used in relations
 between objects, just like normal `datatypes`.
 
+## Definition of links
+Podio offers a templated `Link` class ([see here for more details](links.md))
+that allows to link two arbitrary datatypes without having to introduce a
+`OneToOneRelation` or `OneToManyRelation` inside the corresponding datatypes. In
+order to keep the full definition of a datamodel in the YAML file it is possible
+to declare `links` in the YAML file:
+
+```yaml
+ links:
+ ExampleLink:
+ Description: "A link between two (podio generated) objects"
+ Author: "It could be you"
+ From: ExampleHit
+ To: TypeWithEnergy
+```
+
+This definition will yield the following typedefs
+```cpp
+using ExampleLinkCollection = podio::LinkCollection<ExampleHit, TypeWithEnergy>;
+
+using ExampleLink = typename ExampleLinkCollection::value_type;
+// this is equivalent to
+// using ExampleLink = podio::Link<ExampleHit, TypeWithEnergy>;
+
+using MutableExampleLink = typename ExampleLinkCollection::mutable_type;
+// this is equivalent to
+// using MutableExampleLink = podio::MutableLink<ExampleHit, TypeWithEnergy>;
+```
+
+additionally, this will generate the necessary code to enable I/O for this link
+type.
+
 ### Assigning to interface types
 
 Interface types support the same functionality as normal (immutable) datatypes.

doc/index.rst

@@ -15,6 +15,7 @@ Welcome to PODIO's documentation!
  examples.md
  frame.md
  userdata.md
+ links.md
  storage_details.md
  cmake.md
  advanced_topics.md

doc/links.md

@@ -45,12 +45,16 @@ For a more detailed explanation of the internals and the actual implementation
 see [the implementation details](#implementation-details).
 
 ## How to use `Link`s
-Using `Link`s is quite simple. In line with other datatypes that are
-generated by podio all the functionality can be gained by including the
-corresponding `Collection` header. After that it is generally recommended to
-introduce a type alias for easier usage. **As a general rule `Links` need
-to be declared with the default (immutable) types.** Trying to instantiate them
-with `Mutable` types will result in a compilation error.
+Using `Link`s is quite simple. The most straight forward way is to simply
+declare them as part of the datamodel, [as described
+here](datamodel_syntax.md#definition-of-links). That will result in code
+generation that effectiely does what is described here. However, it's not
+strictly necessary to do that in case non-generated code is preferred. In line
+with other datatypes that are generated by podio all the functionality can be
+gained by including the corresponding `Collection` header. After that it is
+generally recommended to introduce a type alias for easier usage. **As a general
+rule `Links` need to be declared with the default (immutable) types.** Trying to
+instantiate them with `Mutable` types will result in a compilation error.
 
 ```cpp
 #include "podio/LinkCollection.h"

doc/templates.md

@@ -26,19 +26,22 @@ Note that some of the information below will only apply to either of these gener
 Currently PODIO loads templates that are placed in [`<prefix>/python/templates`](/python/templates).
 They are broadly split along the classes that are generated for each datatype or component from the EDM definition:
 
-| template file(s) | content | generated file(s) |
-|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
-| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
-| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
-| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
-| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
-| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
-| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
-| `datamodel.h.jinja2` | The *full datamodel header* that includes everything of a generated EDM (via including all generated `Collections`). | `[<package>]/<package>.h` |
-| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
-| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
-| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia | `[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
-| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name | `[<package>/]<package-name>.jl` |
+| template file(s) | content | generated file(s) |
+|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
+| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
+| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
+| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
+| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
+| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
+| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
+| `datamodel.h.jinja2` | The *full datamodel header* that includes everything of a generated EDM (via including all generated `Collections`). | `[<package>]/<package>.h` |
+| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
+| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
+| `LinkCollection.h.jinja2` | The header that is generated for each *Link* containing effectively typedefs only | |
+| `DatamodelLinksSIOBlock.cc.jinja2` | The .cc file that is necessary for enabling SIO based I/O for *Link*s | |
+| `DatamodelLinks.cc.jinja2` | The global .cc file that is necessary to enable I/O for all *Link*s | |
+| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia | `[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
+| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name | `[<package>/]<package-name>.jl` |
 
 
 The presence of a `[<package>]` subdirectory for the header files is controlled by the `includeSubfolder` option in the yaml definition file.

include/podio/LinkCollection.h

@@ -8,30 +8,17 @@
  #define PODIO_ENABLE_SIO 0
 #endif
 
-/// Macro for registering links at the CollectionBufferFactory by injecting the
-/// corresponding buffer creation function.
-#define PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT) \
+/// Main macro for declaring links. Takes care of registering the necessary
+/// buffer creation functionality with the CollectionBufferFactory.
+#define PODIO_DECLARE_LINK(FromT, ToT)  \
  const static auto PODIO_PP_CONCAT(REGISTERED_LINK_, __COUNTER__) = \
  podio::detail::registerLinkCollection<FromT, ToT>(podio::detail::linkCollTypeName<FromT, ToT>());
 
-/// Macro for registering the necessary SIOBlock for a Link with the SIOBlock factory
-#define PODIO_REGISTER_LINK_SIOFACTORY(FromT, ToT) \
- const static auto PODIO_PP_CONCAT(LINK_SIO_BLOCK_, __COUNTER__) = podio::LinkSIOBlock<FromT, ToT>{};
-
 #if PODIO_ENABLE_SIO && __has_include("podio/detail/LinkSIOBlock.h")
- #include "podio/detail/LinkSIOBlock.h"
- /// Main macro for declaring links. Takes care of the following things:
- /// - Registering the necessary buffer creation functionality with the
- /// CollectionBufferFactory.
- /// - Registering the necessary SIOBlock with the SIOBlock factory
- #define PODIO_DECLARE_LINK(FromT, ToT) \
- PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT) \
- PODIO_REGISTER_LINK_SIOFACTORY(FromT, ToT)
-#else
- /// Main macro for declaring links. Takes care of the following things:
- /// - Registering the necessary buffer creation functionality with the
- /// CollectionBufferFactory.
- #define PODIO_DECLARE_LINK(FromT, ToT) PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT)
+ #include <podio/detail/LinkSIOBlock.h>
+ /// Macro for registering the necessary SIOBlock for a Link with the SIOBlock factory
+ #define PODIO_DECLARE_LINK_SIO(FromT, ToT) \
+ const static auto PODIO_PP_CONCAT(LINK_SIO_BLOCK_, __COUNTER__) = podio::LinkSIOBlock<FromT, ToT>{};
 #endif
 
 #endif // PODIO_LINKCOLLECTION_H

python/podio_gen/cpp_generator.py

@@ -91,14 +91,16 @@ def pre_process(self):
 
  return {}
 
- def post_process(self, _):
+ def post_process(self, datamodel):
  """Do the cpp specific post processing"""
  self._write_edm_def_file()
 
  if "ROOT" in self.io_handlers:
  self._prepare_iorules()
  self._create_selection_xml()
 
+ if datamodel["links"]:
+ self._write_links_registration_file(datamodel["links"])
  self._write_all_collections_header()
  self._write_cmake_lists_file()
 
@@ -207,6 +209,23 @@ def do_process_interface(self, _, interface):
  self._fill_templates("Interface", interface)
  return interface
 
+ def do_process_link(self, _, link):
+ """Process a link definition and generat the necessary code"""
+ link["include_types"] = []
+ for rel in ("From", "To"):
+ rel_type = link[rel]
+ include_header = f"{rel_type.bare_type}Collection"
+ if self._is_interface(rel_type.full_type):
+ # Interfaces do not have a Collection header
+ include_header = rel_type.bare_type
+ link["include_types"].append(
+ self._build_include_for_class(
+ include_header, self._needs_include(rel_type.full_type)
+ )
+ )
+ self._fill_templates("LinkCollection", link)
+ return link
+
  def print_report(self):
  """Print a summary report about the generated code"""
  if not self.verbose:
@@ -506,8 +525,10 @@ def _write_list(name, target_folder, files, comment):
 
  def _write_all_collections_header(self):
  """Write a header file that includes all collection headers"""
-
- collection_files = (x.split("::")[-1] + "Collection.h" for x in self.datamodel.datatypes)
+ collection_files = (
+ x.split("::")[-1] + "Collection.h"
+ for x in list(self.datamodel.datatypes.keys()) + list(self.datamodel.links.keys())
+ )
  self._write_file(
  os.path.join(self.install_dir, self.package_name, f"{self.package_name}.h"),
  self._eval_template(
@@ -520,6 +541,19 @@ def _write_all_collections_header(self):
  ),
  )
 
+ def _write_links_registration_file(self, links):
+ """Write a .cc file that registers all the link collections that were
+ defined with this datamodel"""
+ link_data = {"links": links, "incfolder": self.incfolder}
+ self._write_file(
+ "DatamodelLinks.cc",
+ self._eval_template("DatamodelLinks.cc.jinja2", link_data),
+ )
+ self._write_file(
+ "DatamodelLinkSIOBlock.cc",
+ self._eval_template("DatamodelLinksSIOBlock.cc.jinja2", link_data),
+ )
+
  def _write_edm_def_file(self):
  """Write the edm definition to a compile time string"""
  model_encoder = DataModelJSONEncoder()

python/podio_gen/generator_base.py

@@ -96,15 +96,21 @@ class ClassGeneratorBaseMixin:
  list. This function also has to take care of filling the
  necessary templates!
 
+ do_process_link(name: str, link: dict): do some language specific processing
+ for a link type, populating the link dictionary further.
+ When called only the "class" key will be populated. Return a
+ dictionary or None. If None, this will not be put into the
+ "links" list. This function also has to take care of filling
+ the necessary templates!
+
  post_process(datamodel: dict): do some global post processing for which all
  components and datatypes need to have been processed already.
  Gets called with the dictionary that has been created in
  pre_process and filled during the processing. The process
  components and datatypes are accessible via the "components",
- "datatypes" and "interfaces" keys respectively.
+ "datatypes", "interfaces" and "links" keys respectively.
 
  print_report(): prints a report summarizing what has been generated
-
  """
 
  def __init__(
@@ -154,6 +160,7 @@ def process(self):
  datamodel["components"] = []
  datamodel["datatypes"] = []
  datamodel["interfaces"] = []
+ datamodel["links"] = []
 
  for name, component in self.datamodel.components.items():
  comp = self._process_component(name, component)
@@ -170,6 +177,11 @@ def process(self):
  if interf is not None:
  datamodel["interfaces"].append(interf)
 
+ for name, link in self.datamodel.links.items():
+ proc_link = self._process_link(name, link)
+ if proc_link is not None:
+ datamodel["links"].append(proc_link)
+
  self.post_process(datamodel)
  if self.verbose:
  self.print_report()
@@ -200,6 +212,13 @@ def _process_interface(self, name, interface):
 
  return self.do_process_interface(name, interface)
 
+ def _process_link(self, name, link):
+ """Process a single link definition into a dictionary that can be used
+ in jinja2 templates and return that"""
+ link = deepcopy(link)
+ link["class"] = DataType(name)
+ return self.do_process_link(name, link)
+
  @staticmethod
  def _get_filenames_templates(template_base, name):
  """Get the list of output filenames and corresponding template names"""
@@ -217,6 +236,7 @@ def get_fn_format(tmpl):
  "Collection": "Collection",
  "CollectionData": "CollectionData",
  "MutableStruct": "Struct",
+ "LinkCollection": "Collection",
  }
 
  return f'{prefix.get(tmpl, "")}{{name}}{postfix.get(tmpl, "")}.{{end}}'
@@ -227,6 +247,8 @@ def get_fn_format(tmpl):
  "Interface": ("h",),
  "MutableStruct": ("jl",),
  "ParentModule": ("jl",),
+ "LinkCollection": ("h",),
+ "DatamodelLinks": ("cc",),
  }.get(template_base, ("h", "cc"))
 
  fn_templates = []