You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current implementation of add_function() is very simple. It just stores the function in an internal dict. This has several drawbacks:
it is not persistent
it is not shared between processes or sessions (which might be needed with OTEAPI services running on a distributed system)
it stores information outside the triplestore
I can think about two solutions:
Pickle the function and store the pickle in the triplestore. The pickle module in the standard library will not work, since it requires the function to be available when unpickled. But dill will work for most functions, except for functions with recursive inner functions. However, there is an article about how to also solve that case, see https://medium.com/@greyboi/serialising-all-the-functions-in-python-cd880a63b591.
Identify the function with a set of string properties:
PyPI package name (a string in the same form as requirements.txt files including version requirements), optional
package name, optional
module name
function name
These can easily be stored in the triplestore. To get the conversion function, you can give module and package names as arguments to importlib.import_module() and then fetch the function with getattr() on the return module object. The PyPI package name may be useful if we want to hint the user about a missing package or to dynamically install it if it is missing. Automatic installation raises some security concerns, so if implemented, this feature should probably be explicit enabled.
# Description:
Closes#128
See #128 for design and reasoning behind this PR. Needed for using
mapping functions in OTEAPI pipelines.
## Type of change:
- [ ] Bug fix.
- [x] New feature.
- [ ] Documentation update.
- [ ] Testing.
The current implementation of
add_function()
is very simple. It just stores the function in an internal dict. This has several drawbacks:I can think about two solutions:
Pickle the function and store the pickle in the triplestore. The
pickle
module in the standard library will not work, since it requires the function to be available when unpickled. Butdill
will work for most functions, except for functions with recursive inner functions. However, there is an article about how to also solve that case, see https://medium.com/@greyboi/serialising-all-the-functions-in-python-cd880a63b591.Identify the function with a set of string properties:
These can easily be stored in the triplestore. To get the conversion function, you can give module and package names as arguments to
importlib.import_module()
and then fetch the function withgetattr()
on the return module object. The PyPI package name may be useful if we want to hint the user about a missing package or to dynamically install it if it is missing. Automatic installation raises some security concerns, so if implemented, this feature should probably be explicit enabled.This approach has already been implemented in the
convert
strategy inoteapi-dlite
and seems to work well. See https://github.com/EMMC-ASBL/oteapi-dlite/blob/convert-function-strategy/oteapi_dlite/strategies/convert.py#L57.Personally I prefer the second option, since it is simple and clean and stores no byte code that might result in portability issues.
The text was updated successfully, but these errors were encountered: