Prototype multi-version native graph store

[aionbot]

Issue created by jeff-aion (on Friday Sep 28, 2018 at 16:06 GMT)

Create an implementation of IObjectGraphStore which acts as a native graph store (not built on a key-value store). The complexity of this is in the multi-version support (which will also require the definition of that relationship with IObjectGraphStore). Otherwise, it should be relatively straight-forward to implement this on top of MappedByteBuffer.

[aionbot]

Comment by jeff-aion (on Friday Oct 26, 2018 at 20:36 GMT)

Looking further into MappedByteBuffer, there are some problems with it which may make it not appropriate for this usage (which is odd, since large files is documented as its point - they just can't be too large or numerous):

• No explicit unmap support. While the reasons to avoid this are somewhat understandable, mapped file regions always require some amount of "be careful", so it would be nice to see. As a consequence, it is possible for the mapped files to clutter the address space even when they aren't used (keeping disk resources alive and potentially blocking deletes, on Windows).
• There are limits around the mapping such that only 2 GiB mappings (no greater than Integer.MAX_VALUE) are possible, which would require that we build a paging system to handle how to address fragments of large files (unless we convince ourselves that it would be unreasonably expensive for a single DApp to ever require 2 GiB of data, even including all accessible versions)

An alternative may be to use JNA to access the LibC mmap/unmap calls, directly (some discussion of that idea found here https://stackoverflow.com/questions/15901832/how-to-memory-map-mmap-a-linux-block-device-e-g-dev-sdb-in-java ). The problem is that this substantially limits our portability.

[aionbot]

Comment by jeff-aion (on Monday Nov 19, 2018 at 14:40 GMT)

Moving persistence-related concerns to Beta1. We want this done well before TestNet, but it doesn't gate the actual Beta release since the key-value back-end technically works.

[aionbot]

Comment by jeff-aion (on Thursday Nov 29, 2018 at 19:47 GMT)

Moving to future since, although I suspect this will be very important, it doesn't technically block anything.