Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple definitions at link time #38

Open
aminiussi opened this issue Jun 7, 2022 · 13 comments
Open

Multiple definitions at link time #38

aminiussi opened this issue Jun 7, 2022 · 13 comments

Comments

@aminiussi
Copy link

Hi,

I have an object model using Kokkos views I want to export to python using pybind11.
The export code is in fargOCApy library. So I added the following link directive in my CMake files:

pybind11_add_module(fargOCApy
  fargOCApy.cpp
  exportDisk.cpp
  exportEnvironment.cpp
  exportScalarField.cpp
  exportSimulation.cpp
  )

target_link_libraries(fargOCApy PUBLIC fargoseqrt libpykokkos-core)
set_target_properties(fargOCApy PROPERTIES
  POSITION_INDEPENDENT_CODE 1
  COMPILE_DEFINITIONS FARGO_SEQ
  )

fargoseqrt contains the object model.

If I do that, the link fails with:

../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o: In function `generate_pool_variants(pybind11::module_&)':
/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/pool_variants/pools.cpp:11: multiple definition of `generate_pool_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/pool_variants/pools.cpp.o:/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/pool_variants/pools.cpp:11: first defined here
../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o: In function `generate_atomic_variants(pybind11::module_&)':
/work/aminiussi/fargo/pybind11/external/pykokkos-base/src/variants/atomics.cpp:47: multiple definition of `generate_atomic_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/atomics.cpp.o:/work/aminiussi/fargo/pybind11/external/pykokkos-base/src/variants/atomics.cpp:47: first defined here
../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o: In function `std::_Vector_base<Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void>, std::allocator<Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void> > >::_M_deallocate(Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void>*, unsigned long)':
/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/variants/views.cpp:42: multiple definition of `generate_view_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/views.cpp.o:/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/variants/views.cpp:42: first defined here
make[2]: *** [src/fargOCApy/fargOCApy.cpython-36m-x86_64-linux-gnu.so] Error 1

A verbose build indicates that the pool.o object does indeed appears twice on the link command:

cd /work/aminiussi/fargo/pybind11/dbg/src/fargOCApy && /panfs/panasas/softs/occigen/tools/cmake/3.16.1/bin/cmake -E cmake_link_script CMakeFiles/fargOCApy.dir/link.txt --verbose=1
/opt/software/common/intel/compilers_and_libraries_2019.4.243/linux/bin/intel64/icpc -fPIC  -use-intel-optimized-headers -shared-intel -wd9 -qopenmp -g -DKOKKOS_DEPENDENCE -fopenmp -shared  -o fargOCApy.cpython-36m-x86_64-linux-gnu.so CMakeFiles/fargOCApy.dir/fargOCApy.cpp.o CMakeFiles/fargOCApy.dir/exportDisk.cpp.o CMakeFiles/fargOCApy.dir/exportEnvironment.cpp.o CMakeFiles/fargOCApy.dir/exportScalarField.cpp.o CMakeFiles/fargOCApy.dir/exportSimulation.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/pool_variants/pools.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/atomics.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/views.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/libpykokkos.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/backend_version.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/enumeration.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/available.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/common.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/tools.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o   -L/panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib  -Wl,-rpath,/panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib:/work/aminiussi/fargo/pybind11/dbg/src:/work/aminiussi/fargo/pybind11/dbg/src/noopmpi:/panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib:/work/aminiussi/fargo/pybind11/dbg/src/symba7:/work/aminiussi/fargo/pybind11/dbg/src/swift:/work/aminiussi/fargo/pybind11/dbg/external/kokkos/containers/src:/work/aminiussi/fargo/pybind11/dbg/external/kokkos/core/src ../libfargoseqrt.so ../noopmpi/libnoopmpi.so /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_program_options.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_chrono.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_filesystem.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_system.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_regex.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib/libhdf5_cpp.so /panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib/libhdf5.so /usr/lib64/libz.so /usr/lib64/libdl.so /usr/lib64/libm.so -lm ../symba7/libsymba7seq.so ../swift/libswift.so ../../external/kokkos/containers/src/libkokkoscontainers.so.3.6.00 ../../external/kokkos/core/src/libkokkoscore.so.3.6.00 /usr/lib64/libdl.so

If I do not link with libpykokkos-core, the python test script will fail, as it won't find the Kokkos views binding.

Am I doing something wrong ?

Thx

@jrmadsen
Copy link
Contributor

jrmadsen commented Jun 30, 2022

I do not understand why you want to link to libpykokkos-core in the first place. Install libpykokkos-core alongside your code. Link to Kokkos::Kokkos and use Kokkos views in your code and as long as there is an import kokkos somewhere (which you could do in your bindings via py::module::import("kokkos")), it should "just work".

@jrmadsen
Copy link
Contributor

jrmadsen commented Jun 30, 2022

If I do not link with libpykokkos-core, the python test script will fail, as it won't find the Kokkos views binding.

Do you have an import kokkos somewhere in the python test script?

@aminiussi
Copy link
Author

Hi,

I installed libkkokos-core as a submodule:

aminiussi@login1:/work/aminiussi/fargo/pybind11/rel$ ls external/pykokkos-base/kokkos/
__init__.py  libpykokkos.cpython-36m-x86_64-linux-gnu.so  __pycache__  test  utility.py
aminiussi@login1:/work/aminiussi/fargo/pybind11/rel$ 

At the begining of th test script:

import sys
sys.path.append("/work/aminiussi/fargo/pybind11/rel/src/fargOCApy")
sys.path.append("/work/aminiussi/fargo/pybind11/rel/external/pykokkos-base")

import argparse
import kokkos
import fargOCApy as fargo
[...]

but then the test script will fail with:

63: Traceback (most recent call last):
63:   File "/work/aminiussi/fargo/pybind11/rel/test/fargocapy/explore_seq.py", line 52, in <module>
63:     vradial = simulation.disk.velocity.radial.field
63: TypeError: Unable to convert function return value to a Python type! The signature was
63: 	(arg0: fargOCApy.ScalarField) -> Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>
1/1 Test #63: py_explore_seq ...................***Failed    0.26 sec
 

So maybe I'm missing something in the instantiation chain.

The py11bind based export code is:

namespace py11 {
    void exportScalarField(py::module_& m) {
      py::class_<ScalarField,  std::shared_ptr<ScalarField>>(m, "ScalarField")
        .def_property_readonly("field",
                               [](ScalarField const& v) {
                                 return Kokkos::Experimental::as_python_type(v.data());
                               },
                               "return a view copy",
                               py::return_value_policy::copy)
        .def("__repr__",
             [](ScalarField const& d) {
               return "fargOCApy.ScalarField";
             });
    }

with v.data() returning a Kokko::View<double***> so it looks like some code related with Kokko::View<double***> wasn't instantiated.

@aminiussi
Copy link
Author

Hi
Any hint ?
Thanks

@aminiussi
Copy link
Author

with v.data() returning a Kokko::View<double***> so it looks like some code related with Kokko::View<double***> wasn't instantiated.

Actually, v.data() returns a Kokko::View<double const***>, when I run the test, I have message:

TypeError: Unregistered type : Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>

Is there a way to get the list of registered types ?

@aminiussi
Copy link
Author

Ok so the problem is with the const

Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>

If I change the code to return a Kokkos::View<double***, Kokkos::LayoutRight, Kokkos::HostSpace> it works just fine.

Is this an expected behavior ?

@jrmadsen
Copy link
Contributor

Yes, they are different types and bindings are not built for const data types.

@aminiussi
Copy link
Author

Is there a documented way to explicitly select/instantiate (with cmake help?) the type to instantiate ?

@aminiussi
Copy link
Author

Also, now, when in release mode on linux, pykokkos bulid fails with:

cd /gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/pykokkos-base && /gpfs7kro/gpfslocalsup/spack_soft/cmake/3.21.3/gcc-8.4.1-x74fppdpetkjc5r4jgzwielde7c7o2w3/bin/cmake -E cmake_link_script CMakeFiles/libpykokkos.dir/link.txt --verbose=1
/gpfswork/rech/oth/roth005/fargo/vanilla/external/kokkos/bin/kokkos_launch_compiler /gpfswork/rech/oth/roth005/fargo/vanilla/external/kokkos/bin/nvcc_wrapper /usr/bin/g++ /usr/bin/g++ -fPIC  -fopenmp -O3 -DNDEBUG -flto -fno-fat-lto-objects -DKOKKOS_DEPENDENCE -fopenmp -arch=sm_70 -shared  -o kokkos/libpykokkos.cpython-36m-x86_64-linux-gnu.so CMakeFiles/libpykokkos-core.dir/src/libpykokkos.cpp.o CMakeFiles/libpykokkos-core.dir/src/backend_version.cpp.o CMakeFiles/libpykokkos-core.dir/src/enumeration.cpp.o CMakeFiles/libpykokkos-core.dir/src/available.cpp.o CMakeFiles/libpykokkos-core.dir/src/common.cpp.o CMakeFiles/libpykokkos-core.dir/src/tools.cpp.o CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o src/pool_variants/CMakeFiles/libpykokkos-pool-variants.dir/xorshift64_pool.cpp.o src/pool_variants/CMakeFiles/libpykokkos-pool-variants.dir/xorshift1024_pool.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float64_left_managed.cpp.o  -Wl,-rpath,/gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/kokkos/containers/src:/gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/kokkos/core/src: ../kokkos/containers/src/libkokkoscontainers.so.3.6.00 ../kokkos/core/src/libkokkoscore.so.3.6.00 /gpfslocalsys/cuda/11.2/lib64/stubs/libcuda.so /gpfslocalsys/cuda/11.2/lib64/libcudart.so /usr/lib64/libdl.so 
/tmp/ccaFaZFz.s: Assembler messages:
/tmp/ccaFaZFz.s:43: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:1264: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:2485: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:3706: Error: symbol `fatbinData' is already defined
...
lto-wrapper: fatal error: /usr/bin/g++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [external/pykokkos-base/CMakeFiles/libpykokkos.dir/build.make:172: external/pykokkos-base/kokkos/libpykokkos.cpython-36m-x86_64-linux-gnu.so] Error 1
make[2]: Leaving directory '/gpfsdswork/projects/rech/oth/roth005/fargo/vanilla/gpu/rel'
make[1]: *** [CMakeFiles/Makefile2:2469: external/pykokkos-base/CMakeFiles/libpykokkos.dir/all] Error 2
make[1]: Leaving directory '/gpfsdswork/projects/rech/oth/roth005/fargo/vanilla/gpu/rel'
make: *** [Makefile:146: all] Error 2

@aminiussi
Copy link
Author

So, this one seems to be fixed be setting ENABLE_THIN_LTO to Off.

What is the rationale for setting it On by default ? it is not the default for pybind11 in general. Does it really improve performances that much ?

Also, I could not see it in the documentation.

Thanks

@jrmadsen
Copy link
Contributor

So, this one seems to be fixed be setting ENABLE_THIN_LTO to Off.

What is the rationale for setting it On by default ?

It's not.

ADD_OPTION(ENABLE_THIN_LTO "Pass THIN_LTO to pybind11_add_module instead of NO_EXTRAS" OFF)

it is not the default for pybind11 in general.

Actually it is. Pybind11 enables LTO by default when you use pybind11_add_module. You have to pass NO_EXTRAS to disable it.

Does it really improve performances that much ?

AFAIR, the real benefit is the reduced size of the binary.

@jrmadsen
Copy link
Contributor

Is there a documented way to explicitly select/instantiate (with cmake help?) the type to instantiate ?

No. Providing instantiations for all possible const-qualified types would be a massively unnecessary compilation overhead. And Python doesn't have a concept of const data so having const data wouldn't even translate. You just need a simple template metaprogramming wrapper to provide you with the view type with the const remove and do a static cast.

@jrmadsen
Copy link
Contributor

jrmadsen commented Aug 18, 2022

#include <KokkosExp_InterOp.hpp> // provides Kokkos::Experimental::python_view_type

// provide an additional layer which removes the const from the data type
template <typename ViewT>
struct python_view_type;

template <template <typename...> class ViewT, typename DataT, typename... ExtraT>
struct python_view_type<ViewT<DataT, ExtraT....>>
{
    // key part: 
    //     std::remove_const_t<DataT>
    using type = Kokkos::Experimental::python_view_type_t<ViewT<std::remove_const_t<DataT>, ExtraT...>>;
};

template <typename ViewT>
using python_view_type_t = typename python_view_type<ViewT>::type;

template <typename ViewT>
inline auto as_python_type(ViewT _view)
{
    return static_cast<python_view_type_t<ViewT>>(_view);
}

whenever your bindings return a view to python, e.g. given Kokkos::View<const double***> myview:

m.def("foo", []() {
    // does something and returns view
    return myview;
}, "Returns a view");

you just wrap as_python_type around the returned view, e.g.:

m.def("foo", []() {
    // does something and returns view without const
    return as_python_type(myview);
}, "Returns a view");

it is pretty much the same as doing a const_cast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants