Add verification on mergeSpans

[ahrtr]

It might be helpful to detect issue such as #705

cc @fuweid @tjungblu

[tjungblu]

/lgtm

thanks for adding those, let's see if we can find some triggers of this

[tjungblu]

this is highly off topic here, but I wonder if we can also implement a much simpler freelist with plain bitsets. We would have a much more compact in-memory representation and would get unions/intersections for free.

Had some good experience with github.com/bits-and-blooms/bitset recently while creating that defrag/page usage video.

[ahrtr]

this is highly off topic here, but I wonder if we can also implement a much simpler freelist with plain bitsets. We would have a much more compact in-memory representation and would get unions/intersections for free.

Had some good experience with github.com/bits-and-blooms/bitset recently while creating that defrag/page usage video.

I am not satisfied with the bbolt freelist management for a long time. I have been thinking how to refactor it and improve the situation.

I am open to any idea on this and happy to discuss it further. Will take a look at github.com/bits-and-blooms/bitset later.

[ahrtr]

I am not satisfied with the bbolt freelist management for a long time

There isn't a clear interface and hard to understand. I am thinking to

• define a clear interface & clarify the behavior.
• try to decouple it with the bbolt;
• add more dedicated test for the freelist management

[tjungblu]

yep, that would also be my first course of action. Pull the two implementations under a common interface, then have proper testing for those.

[ahrtr]

github.com/bits-and-blooms/bitset

I just had a quick look at the project. It seems not quite related to our usage o freelist management. It manages a mapping between non-negative integers and boolean values; while bbolt freelist management just needs to remember a list of free page IDs.

If you still think it might be helpful or related, please feel free to raise a separate ticket to discuss. thx

[tjungblu]

If you still think it might be helpful or related, please feel free to raise a separate ticket to discuss. thx

will do. You can use a bitset to map from PageId->Free/Used. Since you're packing 64 bools into one int64, you have a much lower memory/file usage. Finding continuous free blocks is also cheaper, since you can just bitshift along the int64.