Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

abort: packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() #24

Closed
smoe opened this issue Jul 18, 2020 · 16 comments · Fixed by #25

Comments

@smoe
Copy link

smoe commented Jul 18, 2020

Hello,
ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1.
Anything you want me to do about it?

Cheers,
Steffen

@smoe
Copy link
Author

smoe commented Jul 18, 2020

First invocation:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:252ms
rank:331ms
wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

and in gdb:

Core was generated by `./obj-x86_64-linux-gnu/wm_string'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f08b36ef55b in __GI_abort () at abort.c:79
#2  0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0", 
    file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
#3  0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, 
    function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
    at assert.c:101
#4  0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=<optimized out>, rank=0, c=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:1058
#5  0x000056266528c474 in speed_select (num=100000, num_of_alphabet=<optimized out>) at ./wm_string.cpp:74
#6  0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
#7  0x000056266528aa2e in main () at ./wm_string.cpp:394

This is very nicely reproducible.

@ekg
Copy link
Contributor

ekg commented Jul 18, 2020 via email

@ekg
Copy link
Contributor

ekg commented Jul 18, 2020 via email

@smoe
Copy link
Author

smoe commented Jul 18, 2020

That is what I presume since I requested a pull from vgteam/DYNAMIC.

@nicolaprezza
Copy link
Member

Hello,
ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1.
Anything you want me to do about it?

Cheers,
Steffen

I thought I had solved this bug but apparently it's alive and well. I'll look into it next week.

nicolaprezza added a commit that referenced this issue Jul 22, 2020
@nicolaprezza
Copy link
Member

Hi, I solved the first part of the issue:

Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.

Actually, the code was correct and only the assert had a bug : -)

@ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.

@ekg
Copy link
Contributor

ekg commented Jul 22, 2020 via email

@smoe
Copy link
Author

smoe commented Jul 22, 2020

There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?

@ekg
Copy link
Contributor

ekg commented Jul 22, 2020 via email

@smoe
Copy link
Author

smoe commented Jul 22, 2020

All dependencies of DYNAMIC are in Debian/Ubuntu already - I guess this would be rather straight-forward to implement. Ping me if you need help.

@ekg
Copy link
Contributor

ekg commented Jul 22, 2020

@nicolaprezza I'm not able to reproduce the problem with wm_string. It's at least not happened in the 20-30 runs that I've done (running it in a loop and watching...)

@smoe
Copy link
Author

smoe commented Jul 22, 2020

Latest checkout:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:229ms
rank:293ms
wm_string: /home/moeller/git/med-team/dynamic/a/DYNAMIC/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

Maybe because of other compiler settings?

/usr/bin/cmake -E cmake_link_script CMakeFiles/wm_string.dir/link.txt --verbose=1
/usr/local/bin/c++  -g -O2 -fdebug-prefix-map=/home/moeller/git/med-team/dynamic/a/DYNAMIC=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --std=c++11  -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -rdynamic CMakeFiles/wm_string.dir/wm_string.cpp.o  -o wm_string 

The benchmark seems stable now. Would you have other ideas what I should test over here?

./obj-x86_64-linux-gnu/cw-bwt /etc/passwd /tmp/bla.txt worked - but not on /proc/cpuinfo
./obj-x86_64-linux-gnu/h0_lz77 /etc/passwd /tmp/bla.txt worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txt worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txtworked

I can get a VM online somewhere for a common playground to investigate the wm_string issue.

@nicolaprezza
Copy link
Member

I can reproduce it. Have you enabled asserts? with cmake, it's simply cmake .. -DCMAKE_BUILD_TYPE=Debug

@ekg
Copy link
Contributor

ekg commented Jul 22, 2020

Thanks :) That's my mistake. I can reproduce it by enabling the assert!

@ekg
Copy link
Contributor

ekg commented Jul 23, 2020

It does look like a possible problem with the input. There is a quirky PRNG being used. I might switch to modern c++ random distributions.

@nicolaprezza
Copy link
Member

Thanks @ekg! now it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants