[WIP] feature : add zero copy support for vector index

[foxspy]

issue: #1031

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: foxspy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [foxspy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@foxspy 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

1. If you're fixing a bug, label it as kind/bug.
2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.61%. Comparing base (3c46f4c) to head (7dc2032).
Report is 292 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1032       +/-   ##
=========================================
+ Coverage      0   73.61%   +73.61%     
=========================================
  Files         0       82       +82     
  Lines         0     7511     +7511     
=========================================
+ Hits          0     5529     +5529     
- Misses        0     1982     +1982     

see 82 files with indirect coverage changes

[alexanderguzhva]

one topic to discuss. Otherwise, lgtm

[alexanderguzhva]

this is a very tricky point. Although, I understand why this was introduced, I'd like to avoid adding it here. Basically, the approach is similar to a std::string_view, which does not have a business to maintain the memory. Similar, here: It is not Index's instance business to maintain the view.

I'm open for discussions.

[alexanderguzhva]

this is a todo for the future: basically, a reader need to instantiate an object. But this is a later business, because it needs to be properly synchronized with the vanilla faiss first.
So, let's keep the change in the way you proposed it.

[foxspy]

Yes, so I put the PR up first; zero copy is needed here mainly for

1. Optimizing memory expansion caused by copying during loading
2. The memory will be managed uniformly through Eyrie later, and the index does not hold data, but only views
3. The implementation of mmap and zero copy can be unified (mmap externally first, load through zero copy)
   If you have any ideas about this, please feel free to communicate; has faiss considered supporting this mode?