Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRIVERS-2926] [PYTHON-4577] BSON Binary Vector Subtype Support #1813

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
245c869
First commit on DRIVERS-2926-BSON-Binary-Vectors
caseyclements Aug 22, 2024
031cd8c
Turns dtype into enum. Adds handling of padding, __eq__. Removal of n…
caseyclements Aug 23, 2024
8d4e8a2
Added docstring and comments
caseyclements Aug 23, 2024
2df0d6b
Changed order of BinaryVector and Binary in bson._ENCODERS to get tes…
caseyclements Aug 23, 2024
315a115
Changed order of BinaryVector and Binary in bson._ENCODERS to get tes…
caseyclements Aug 23, 2024
d74314d
json_util dumps/loads of BinaryVector
caseyclements Aug 23, 2024
27f13c8
Added bson_corpus tests. Needs more, and review of json_util
caseyclements Aug 24, 2024
263f8c7
Removed BinaryVector as separate class. Instead, Binary includes as_v…
caseyclements Sep 12, 2024
f8bcdef
Stop setting _USD_C to False
caseyclements Sep 13, 2024
5435785
mypy fixes
caseyclements Sep 13, 2024
5c4d152
Removed stub vector.json for bson_corpus tests
caseyclements Sep 13, 2024
f86d040
More tests
caseyclements Sep 13, 2024
adcb945
Added description of subtype 9 to bson.Binary docstring
caseyclements Sep 14, 2024
7986cc5
Addressed comments in docstrings.
caseyclements Sep 16, 2024
26b8398
Eased string comparison of exception in xfail in test_bson
caseyclements Sep 16, 2024
28de28a
Updates to docstrings of BinaryVector and BinaryVectorDtype
caseyclements Sep 17, 2024
68235b8
Simplified expected exeption case. Will be refactored with yaml anyway..
caseyclements Sep 17, 2024
e2a1a3c
Added draft of test runner
caseyclements Sep 18, 2024
bf9758a
Added test cases: padding, and overflow
caseyclements Sep 19, 2024
e1590aa
Merge branch 'master' into DRIVERS-2926-BSON-Binary-Vectors
caseyclements Sep 19, 2024
c4c7af7
Cast Path to str
caseyclements Sep 19, 2024
de5a245
Simplified as_vector API
caseyclements Sep 20, 2024
43bcce4
Added test case: list of floats with dtype int8 raises exception
caseyclements Sep 20, 2024
41ee0bb
Set default padding to 0 in test runner
caseyclements Sep 20, 2024
9d52aeb
Updated test_bson for new as_vector API
caseyclements Sep 20, 2024
0d34464
Updated resync-specs.sh to include bson-binary-vector
caseyclements Sep 20, 2024
1d49656
Updated resync-specs.sh and test cases
caseyclements Sep 20, 2024
2af0ca4
Broke tests into 3 files by dtype
caseyclements Sep 20, 2024
c93bae1
Update bson/binary.py
caseyclements Sep 27, 2024
f374b5a
Removed json from test_bson_binary_vector and its jsons
caseyclements Sep 27, 2024
0db9866
Addition of Provision (BETA) specifiers change references to 4.10
caseyclements Sep 30, 2024
0532803
Add references to from_vector() and as_vector()
caseyclements Sep 30, 2024
3edeef6
Add subtype number in changelog
caseyclements Sep 30, 2024
d199597
Raise ValueErrors not AssertionErrors. Bumped from 4.9 to 4.10
caseyclements Sep 30, 2024
abc7cd3
Docstring for as_vector
caseyclements Sep 30, 2024
4550c20
Add slots for BinaryVector
caseyclements Sep 30, 2024
99d44e1
Check subtype before decoding
caseyclements Oct 1, 2024
001636d
Try slots with default padding
caseyclements Oct 1, 2024
637c474
Removed slots arg
caseyclements Oct 1, 2024
2d511f6
Update dataclass
caseyclements Oct 1, 2024
17e1d33
Remove unompressed kwarg from as_vector
caseyclements Oct 1, 2024
ce5f3e3
Changed TypeError to ValueError
caseyclements Oct 1, 2024
edfe972
Updates after removing uncompressed
caseyclements Oct 1, 2024
8aaa2f6
Fixed expected exceptions in invalid test cases
caseyclements Oct 1, 2024
dfb322c
Merge branch 'master' into DRIVERS-2926-BSON-Binary-Vectors
blink1073 Oct 1, 2024
8946daf
padding in now Optional[int] = None
caseyclements Oct 1, 2024
9397129
padding does need to be an integer
caseyclements Oct 1, 2024
913403b
Removed unneeded ugly TYPE_FROM_HEX = {key.value: key for key in Bina…
caseyclements Oct 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions bson/binary.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,9 +195,9 @@ class UuidRepresentation:


VECTOR_SUBTYPE = 9
"""BSON binary subtype for densely packed vector data.
"""**(BETA)** BSON binary subtype for densely packed vector data.

.. versionadded:: 4.9
.. versionadded:: 4.10
"""


Expand All @@ -207,7 +207,7 @@ class UuidRepresentation:


class BinaryVectorDtype(Enum):
"""Datatypes of vector subtype.
"""**(BETA)** Datatypes of vector subtype.

:param FLOAT32: (0x27) Pack list of :class:`float` as float32
:param INT8: (0x03) Pack list of :class:`int` in [-128, 127] as signed int8
Expand All @@ -233,7 +233,7 @@ class BinaryVectorDtype(Enum):

@dataclass
class BinaryVector:
"""Vector of numbers along with metadata for binary interoperability.
"""**(BETA)** Vector of numbers along with metadata for binary interoperability.

:param data: Sequence of numbers representing the mathematical vector.
:param dtype: The data type stored in binary
Expand All @@ -257,7 +257,7 @@ class Binary(bytes):
the difference between what should be considered binary data and
what should be considered a string when we encode to BSON.

Subtype 9 provides a space-efficient representation of 1-dimensional vector data.
**(BETA)** Subtype 9 provides a space-efficient representation of 1-dimensional vector data.
Its data is prepended with two bytes of metadata.
The first (dtype) describes its data type, such as float32 or int8.
The second (padding) prescribes the number of bits to ignore in the final byte.
Expand All @@ -278,8 +278,8 @@ class Binary(bytes):
.. versionchanged:: 3.9
Support any bytes-like type that implements the buffer protocol.

.. versionchanged:: 4.9
Addition of vector subtype.
.. versionchanged:: 4.10
**(BETA)** Addition of vector subtype.
"""

_type_marker = 5
Expand Down Expand Up @@ -405,7 +405,7 @@ def from_vector(
dtype: BinaryVectorDtype,
padding: Optional[int] = 0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional[int] -> int

) -> Binary:
"""Create a BSON :class:`~bson.binary.Binary` of Vector subtype from a list of Numbers.
"""**(BETA)** Create a BSON :class:`~bson.binary.Binary` of Vector subtype from a list of Numbers.

To interpret the representation of the numbers, a data type must be included.
See :class:`~bson.binary.BinaryVectorDtype` for available types and descriptions.
Expand Down Expand Up @@ -435,7 +435,7 @@ def from_vector(
return cls(metadata + data, subtype=VECTOR_SUBTYPE)

def as_vector(self, uncompressed: Optional[bool] = False) -> BinaryVector:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional[bool] -> bool unless uncompressed=None is valid.

"""From the Binary, create a list of numbers, along with dtype and padding.
"""**(BETA)** From the Binary, create a list of numbers, along with dtype and padding.


:param uncompressed: If true, return the true mathematical vector.
Expand Down
5 changes: 5 additions & 0 deletions doc/changelog.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
Changelog
=========

Changes in Version 4.10.0
-------------------------

- Provisional **(BETA)** support for a new Binary BSON subtype used for efficient storage and retrieval of vectors: densely packed arrays of numbers, all of the same type.
blink1073 marked this conversation as resolved.
Show resolved Hide resolved

Changes in Version 4.9.0
-------------------------

Expand Down
Loading