make/standalone does not work on Ubuntu

[rok-cesnovar]

Description

I tried to compile a standalone file with math (make -f math/make/standalone test1) and got the following error:

/usr/bin/ld: /tmp/cc8q4NSu.o: in function `tbb::internal::task_scheduler_observer_v3::~task_scheduler_observer_v3()':
test1.cpp:(.text._ZN3tbb8internal26task_scheduler_observer_v3D2Ev[_ZN3tbb8internal26task_scheduler_observer_v3D5Ev]+0x14): undefined reference to `tbb::internal::task_scheduler_observer_v3::observe(bool)'
/usr/bin/ld: /tmp/cc8q4NSu.o: in function `tbb::internal::task_scheduler_observer_v3::~task_scheduler_observer_v3()':
test1.cpp:(.text._ZN3tbb8internal26task_scheduler_observer_v3D0Ev[_ZN3tbb8internal26task_scheduler_observer_v3D5Ev]+0x18): undefined reference to `tbb::internal::task_scheduler_observer_v3::observe(bool)'
/usr/bin/ld: /tmp/cc8q4NSu.o: in function `__static_initialization_and_destruction_0(int, int) [clone .constprop.534]':
test1.cpp:(.text.startup+0x2175): undefined reference to `tbb::internal::task_scheduler_observer_v3::observe(bool)'
/usr/bin/ld: /tmp/cc8q4NSu.o: in function `tbb::interface6::task_scheduler_observer::~task_scheduler_observer()':
test1.cpp:(.text._ZN3tbb10interface623task_scheduler_observerD2Ev[_ZN3tbb10interface623task_scheduler_observerD5Ev]+0x18): undefined reference to `tbb::internal::task_scheduler_observer_v3::observe(bool)'
/usr/bin/ld: test1.cpp:(.text._ZN3tbb10interface623task_scheduler_observerD2Ev[_ZN3tbb10interface623task_scheduler_observerD5Ev]+0x34): undefined reference to `tbb::internal::task_scheduler_observer_v3::observe(bool)'
/usr/bin/ld: /tmp/cc8q4NSu.o:test1.cpp:(.text._ZN3tbb10interface623task_scheduler_observerD0Ev[_ZN3tbb10interface623task_scheduler_observerD5Ev]+0x18): more undefined references to `tbb::internal::task_scheduler_observer_v3::observe(bool)' follow
collect2: error: ld returned 1 exit status
make: *** [<builtin>: test1] Error 1

Its missing -ltbb flag. If I add

LDLIBS += -ltbb to my make/local it works fine.

`#### Current Version:
v3.0.0

[rok-cesnovar]

@wds15 should this line https://github.com/stan-dev/math/blob/develop/make/compiler_flags#L215 just be

LDLIBS_TBB ?= -ltbb

The other way of solving this is just adding LDLIBS += -ltbb to make/standalone. I can make a PR to fix this,

[wds15]

No, we give the library using an absolute path to the binary in order to ensure that the absolute path is baked into the library.

Let me try this on our Linux box to reproduce the problem.

(we can use LDLIBs, but then I am not anymore sure if the rpath trick works, that's my concern)

[rok-cesnovar]

For the record, I love the standalone feature. It made on-boarding a new masters student, previously unfamiliar with Stan Math, really simple. Well apart from this hiccup, but this is fixable.

[wds15]

Thanks... yeah, I also think that standalone is quite useful (performance testing makefiles for example get a lot simpler).

I tried to reproduce on my MacBook and on our RHEL Linux. It works on both systems just fine.

Can you please

• ensure that make -f math/make/standalone math-libs has been called
• show the actual compiler command being generated from make in order to bake test
• make a ls math/lib/tbb
• Is the TBB installed somewhere else on the system?
• Are there some spaces in some paths?

and post these?

What Ubuntu version are you on? Does this happen on all Ubuntu's?

Thanks.

[rok-cesnovar]

• ensure that make -f math/make/standalone math-libs has been called

Yep, ran make clean-all in math and then
make -f math/make/standalone math-libs
make -f math/make/standalone test1

I do have to note that I tried with clang++ today and that seemed to work. With g++ I get the error above.

• show the actual compiler command being generated from make in order to bake test

g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare   -I /home/rok/Desktop/Stan/math/lib/opencl_2.2.0   -I /home/rok/Desktop/Stan/math/lib/tbb_2019_U8/include -O3  -I /home/rok/Desktop/Stan/math/ -I /home/rok/Desktop/Stan/math/lib/eigen_3.3.3 -I /home/rok/Desktop/Stan/math/lib/boost_1.72.0 -I /home/rok/Desktop/Stan/math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes          -Wl,-L,"/home/rok/Desktop/Stan/math/lib/tbb" -Wl,-rpath,"/home/rok/Desktop/Stan/math/lib/tbb" /home/rok/Desktop/Stan/math/lib/tbb/libtbb.so.2 /home/rok/Desktop/Stan/math/lib/sundials_4.1.0/lib/libsundials_nvecserial.a /home/rok/Desktop/Stan/math/lib/sundials_4.1.0/lib/libsundials_cvodes.a /home/rok/Desktop/Stan/math/lib/sundials_4.1.0/lib/libsundials_idas.a /home/rok/Desktop/Stan/math/lib/sundials_4.1.0/lib/libsundials_kinsol.a   test1.cpp       -lOpenCL   -o test1

I have OpenCL on in this case but its the same issue without it.

• make a ls math/lib/tbb

arena.d                    condition_variable.o     market.d              rml_tbb.o                   tbb_main.o
arena.o                    critical_section.d       market.o              scheduler.d                 tbb-make-check
backend.d                  critical_section.o       mutex.d               scheduler.o                 tbbmalloc.d
backend.o                  dynamic_link.d           mutex.o               semaphore.d                 tbbmalloc.def
backref.d                  dynamic_link.o           observer_proxy.d      semaphore.o                 tbbmalloc.o
backref.o                  frontend.d               observer_proxy.o      spin_mutex.d                tbbmallocproxy.def
cache_aligned_allocator.d  frontend.o               pipeline.d            spin_mutex.o                tbb_misc.d
cache_aligned_allocator.o  governor.d               pipeline.o            spin_rw_mutex.d             tbb_misc_ex.d
concurrent_hash_map.d      governor.o               private_server.d      spin_rw_mutex.o             tbb_misc_ex.o
concurrent_hash_map.o      itt_notify.d             private_server.o      spin_rw_mutex_v2.d          tbb_misc.o
concurrent_monitor.d       itt_notify_malloc.d      proxy.d               spin_rw_mutex_v2.o          tbb_statistics.d
concurrent_monitor.o       itt_notify_malloc.o      proxy.o               task.d                      tbb_statistics.o
concurrent_queue.d         itt_notify.o             queuing_mutex.d       task_group_context.d        tbb_thread.d
concurrent_queue.o         large_objects.d          queuing_mutex.o       task_group_context.o        tbb_thread.o
concurrent_queue_v2.d      large_objects.o          queuing_rw_mutex.d    task.o                      tbbvars.csh
concurrent_queue_v2.o      libtbbmalloc_proxy.so    queuing_rw_mutex.o    task_v2.d                   tbbvars.sh
concurrent_vector.d        libtbbmalloc_proxy.so.2  reader_writer_lock.d  task_v2.o                   version_string.ver
concurrent_vector.o        libtbbmalloc.so          reader_writer_lock.o  tbb.def                     version_tbb_2019_U8
concurrent_vector_v2.d     libtbbmalloc.so.2        recursive_mutex.d     tbb_function_replacement.d  x86_rtm_rw_mutex.d
concurrent_vector_v2.o     libtbb.so                recursive_mutex.o     tbb_function_replacement.o  x86_rtm_rw_mutex.o
condition_variable.d       libtbb.so.2              rml_tbb.d             tbb_main.d

There is no system TBB install or spaces in the Path. I tried it on Ubuntu 19.04 and 18.10 and the student that discovered this issue tried it on Linux Mint.

[wds15]

This is all odd. Can you please experiment a bit? Maybe it is about the order of things. So could you test if compiling unit tests works (maybe one which uses the TBB). Then please compare the order of the arguments being given to the compiler. Maybe the libtbb.so.2 is in a different location of the call to the compiler which makes the difference.

Thanks. Sorry I am a bit in the dark right now...

[andrjohns]

I had the same issue with g++ and the TBB under linux over in the perf-math repo. I couldnt find a fix and just ended up using clang++ as a workaround

[wds15]

One more thought: replace the libtbb.so.2 filename in the call to g++ with libtbb.so. Maybe g++ is making some assumptions about filename extensions...

[wds15]

Oh... I now have this problem reproducible in a Docker container running Ubuntu 16.04.

So I can debug. Let's see what's the matter here.

[wds15]

Fixed. The order of arguments was messed up. See the PR which is there in a minute.