vineyard v0.16.3

Vineyard v0.16.3 comes with the following changes:

• Improve the vineyard Go SDK and Rust SDK
• Improve the vineyardctl and k8s integration
• Fixes the peak memory usage during compacting CSR in arrow fragment
• Fixes the bug about object ID generation on Arm platform.

Commits

• [41a59381]: Bump up vineyard version to v0.16.3 (Tao He)
• [0c852fd6]: Support WSL1 support where memfd_create returns ENOSYS (function not implemented). (#1519) (Tao He)
• [6d7dfc49]: Improve the varint encoding for edges by avoiding copy and extra memory (#1516) (Tao He)
• [b85a4789]: Use the golang sdk for tracking chunks for best-effort scheduling and add the relevant e2e test (#1508) (Ye Cao)
• [c2d21534]: set seed for random generator (#1517) (Siyuan Zhang)
• [8ccd4980]: fix the deploy backup job failed when set the pv-pvc-spec (#1515) (Ye Cao)
• [ff4e7fb1]: Implements a shrink API on blob to allow resize blob builder (#1514) (Tao He)
• [0ca7ea32]: Implement cluster-info API of Golang SDK and add the vineyardctl get cluster-info subcommand (#1507) (Ye Cao)
• [7a275fcd]: Revert "[Snyk] Security upgrade alpine from latest to 3.18.3 (#1506)" (Tao He)
• [189f0cff]: Embed operator manifests into vineyardctl and install operator from them (#1513) (Ye Cao)
• [dbf1a50a]: Fixes object ID generation on arm platform (#1510) (Tao He)
• [5a4af631]: Unify the log for vineyardctl and find an appropriate port for forwarding (#1505) (Ye Cao)
• [4dc4a98c]: [Snyk] Security upgrade alpine from latest to 3.18.3 (#1506) (Tao He)
• [52f99369]: Implement the rust SDK: client, object & metadata and array as an example data structure (#1504) (Tao He)
• [a7ea4001]: Bump the airflow to v2.6.3 and improve the doc to adapt to the latest vineyardctl. (#1500) (Ye Cao)
• [8e9caed3]: Add the namespace configurations to the podAffinity and delete the legacy '--wait' flag of helm chart (#1502) (Ye Cao)
• [aa29bbc0]: Fixes the behaviour of removing properties (#1496) (Tao He)

vineyard v0.16.2

Vineyard v0.16.2 improve the the vineyardctl and kedro plugin, and fixes several bugs in io adaptors.

Commits

• [f65e18c5]: Add the missing extract tag step for release vineyardctl CI (#1495) (Ye Cao)
• [0d01e161]: Bump up vineyard version to v0.16.2 (Tao He)
• [55eb0e77]: Fixes the io adaptor parameters: the key should be string (#1493) (Tao He)
• [b438f0e7]: Delete the '--wait' flag during the installation of helm chart. (#1492) (Ye Cao)
• [56b82024]: Add a tutorial for data sharing with vineyard on kubernetes via vineyardctl (#1486) (Ye Cao)
• [adc335b2]: Add cloudpickle-s3 dataset to the benchmark (#1489) (Ye Cao)
• [f2137d0b]: Add socket volume and volumesMount to the workload if not exist and support to inject a pod in the vineyardctl (#1488) (Ye Cao)
• [70d32f20]: Unify table/batch schemas when reading chunks from vineyard streams (#1485) (Tao He)
• [540511ea]: Fixes the bug in vineyard docker image generation (#1483) (Tao He)
• [6c9c7de3]: Add the installation doc of vineyardctl and fix typos of vineyard operator doc (#1481) (Ye Cao)
• [d6ec099b]: Add the vineyardctl of mainstream architecture and os to the release page (#1480) (Ye Cao)
• [a10d87e0]: Improve the kedro mlops tutorial: remove duplication and revise docs (#1479) (Tao He)
• [015b1f28]: Fixes the Google Analytics on vineyard pages (#1470) (Tao He)
• [85eff5a8]: Fixes macos wheels building: use a update-to-date version for apache-arrow (#1467) (Tao He)

vineyard v0.16.1

Vineyard v0.16.1 came with many new features:

• The legacy vineyard-cli has been fully integrated into vineyardctl command line tool for a unified user interfaces.
• torch.data and torcharrow has been integrated into vineyard Python SDK for smooth user experience.
• velox has been integrated for data analytics on vineyard dataframe and tables.
• New website has been published to https://v6d.io.

Commits

• [39f463ab]: Bump up vineyard version to v0.16.1 (Tao He)
• [93c65fd7]: Bump up vineyard version to v0.16.0 (Tao He)
• [71d01a58]: Add velox integration on arrays, dataframes and tables (#1466) (Tao He)
• [cb68733c]: Enrich the functionality of vineyard.connect() method (#1465) (Tao He)
• [12d0d933]: Support to list vineyard objects, metadata, and blobs on the local machine and kubernetes cluster (#1464) (Ye Cao)
• [8df9f248]: Improve pytorch data / torcharrow / torchdata integration (#1463) (Tao He)
• [5b248ceb]: Add a chunk_hook support for vineyard.{read/write} in io adaptors (#1462) (Tao He)
• [cf27180f]: Website: Tiny UI Improvements (#1461) (Uchechukwu Obasi)
• [231fea0e]: Website: Create new landing page (#1449) (Uchechukwu Obasi)
• [999bcd80]: Use distroless rather than alpine-glibc as the base image (#1460) (Tao He)
• [f85e6e0b]: Bump up the version of etcd in the vineyardd image to v3.5.9 (#1459) (Ye Cao)
• [5b30c2e9]: Improve the kedro integration documentation (#1456) (Tao He)
• [eb66753e]: Bump google.golang.org/grpc from 1.47.0 to 1.53.0 in /k8s (#1455) (dependabot[bot])
• [643bf909]: Bump up the version of kubernetes-log-export-action to v5 (#1454) (Ye Cao)
• [7b5b9c15]: Try preview docs using netlify (#1453) (Tao He)
• [6d3c35f3]: Bump up the k8s version to v1.25.10 to fix the existing Kubelet bug (#1448) (Ye Cao)
• [0a98a140]: Update the kedro integration performance report. (#1450) (Ye Cao)
• [32b6e08b]: Bump up the k8s to v1.25.0 and address the security issues (#1447) (Ye Cao)

vineyard v0.15.3

vineyard v0.15.3

vineyard v0.15.2

vineyard v0.15.2 is a bugfix release which addressed an issue about creating remote blobs when vineyardd itself is out-of-memory.

Commits

• [5e418b59]: Bump up vineyard version to v0.15.2 (Tao He)
• [56ff90d3]: Fixes remote blob creation error when OOM (#1433) (Tao He)

vineyard v0.15.1

Vineyard v0.15.1 includes the following minor changes:

• Use a customized Buffer() class to avoid requiring apache-arrow in vineyard_client.
• Add an optional argument to control the parallelism of serializer/deserializer.
• Defaults the shared memory limit to not limited.
• Fixes creating remote blobs when there are trailing 0-sized chunks.

Commits

• [31a3e340]: Bump up vineyard version to v0.15.1 (Tao He)
• [75e3dc4e]: Fixes creating remote blobs when there are trailing 0-sized chunks (#1431) (Tao He)
• [da55873d]: Add the camera ready artifacts for the publication (#1432) (Tao He)
• [78b4d72f]: Fix ConcatenateTables() (#1429) (Songqing Zhang)
• [7d647a5c]: Bump guava from 31.0.1-jre to 32.0.0-jre in /packages-java (#1428) (dependabot[bot])
• [50e59370]: Bump guava from 31.0.1-jre to 32.0.0-jre in /java (#1427) (dependabot[bot])
• [988a471b]: Defaults the shared memory limit to not limited (#1425) (Tao He)
• [55a17bb8]: Check in the missing catalog.py file (#1424) (Tao He)
• [765f91fb]: Implements Buffer/MutableBuffer to remove the arrow deps in vineyard client (#1422) (Tao He)
• [4136522c]: Add the framework for hive integration (#1421) (Tao He)
• [86af8dd5]: Support perfect hashmap in VertexMap (#1408) (vegetableysm)
• [2f8f9518]: Fixes warnings with recent gcc/clang (#1417) (Tao He)
• [13ccf1bf]: Be compatible with Chinese locale for gcc outputs. (#1415) (Tao He)
• [195e57a2]: Add an optional argument to control the parallelism of serailizer/deserializer (#1414) (Tao He)
• [67f893f9]: Use add_subdirectory to generate correct target-*.cmake (#1413) (Tao He)
• [3150e4ad]: Remove the nlohmann/json submodule. (#1412) (Tao He)

vineyard v0.15.0

Vineyard v0.15.0 brings the following two key enhancements:

• Enable delta encoding + varint compression for CSR to reduce the memory usage of graph data structure
• Add backup and failure functionality to vineyardctl interfaces.

Commits

• [b3e01845]: Bump up vineyard version to v0.15.0 (Tao He)
• [286a3300]: Fixes a bug in arrow fragment builder for local vertex map (Tao He)
• [1e1b3998]: Fixes the typename generation for fragment types (#1411) (Tao He)
• [e77ed2b5]: Fixes the delta decoding of compact CSR (#1410) (Tao He)
• [9da4c149]: Fixes the exception during resolving path of vineyardd. (#1409) (Tao He)
• [b8930aeb]: Support to deploy the backup and recover job with an existing pvc (#1407) (Ye Cao)
• [b9e5ed73]: Fix a bug to avoid the panic 'interger divide by zero' during scheduling (#1405) (Ye Cao)

vineyard v0.14.8

Vineyard v0.14.8 adds compression support to the RPC client for efficient remote blob transfer and manipulation.

Commits

• [c20acf8d]: Bump up vineyard version to v0.14.8 (Tao He)
• [a2cdee5a]: Improve GetInnerVertex() by querying local fragment only (#1404) (Songqing Zhang)
• [19616c8c]: Replace the limits with the objectIDs in the backup and recover of vineyard cluster. (#1402) (Ye Cao)
• [2c0957bd]: Batch encoding/decoding (with SIMD) of varint CSR (#1403) (Tao He)
• [e8952166]: Upgrade the libgrape-lite submodule (#1401) (Dongze Li)
• [63caa8e9]: Add compression to rpc clients for remote blobs (#1400) (Tao He)
• [d9a30617]: Simplify the template arguments of loader and modularize the builder (#1399) (Tao He)
• [25390d2b]: Add the labelselector for injected resources to bind the vineyard rpc service (#1398) (Ye Cao)

vineyeard v0.14.7

Vineyard v0.14.7 is a bugfix release that address the volume issue in vineyardctl.

Commits

• [ced8892a]: Bump up vineyard version to v0.14.7 (Tao He)
• [991a9393]: Fix the errors of templates when set the volume path of vineyard socket (#1397) (Ye Cao)
• [0c9c16e4]: Implements varint + delta encoding on CSR to optimize the fragment memory usage (#1372) (vegetableysm)
• [a0b6fe3c]: Kedro integration: the vineyardataset, plugin and cli tools (#1287) (Tao He)
• [0b6e36d0]: Add write vertices num to FragmentWriter to adapt the latest GraphAr format (#1395) (Weibin Zeng)
• [51bc511d]: Remove the ctti dependency (#1394) (Tao He)
• [af45fc6b]: Enable cpplint over vineyard-mod files (#1393) (Tao He)
• [e297c80d]: Upload the e2e test of deploying raw backup and recover logs to stoat dev (#1392) (Ye Cao)

vineyard v0.14.6

Vineyard v0.14.6 is a bugfix release which addressed many issues in vineyardctl and vineyard-graph-loader.

Commits

• [b2a81047]: Bump up vineyard version to v0.14.6 (Tao He)
• [1e6ed21f]: Fixes the bug in transfer large buffers over rpc client (#1391) (Tao He)
• [c6be274b]: Support deploying a raw backup and recover job for vineyard cluster via vineyardctl (#1384) (Ye Cao)
• [4113deb7]: Improve the error message about misuses of IPC/RPC client (#1387) (Tao He)
• [e934861a]: Specify the required version of libgrape-lite (#1388) (Tao He)
• [2e218974]: Wait the vineyard deployment for ready while using vineyardctl deploy vineyard-deployment (#1383) (Ye Cao)
• [9ed9e745]: Fixes the codegen script for non-type template parameter (#1382) (Tao He)
• [753a1416]: codegen: support TEMPLATE_NON_TYPE_PARAMETER in templates (#1381) (Tao He)
• [55b6ec88]: Load multiple edges with same label but different src/dst (#1380) (Tao He)
• [d3640e3c]: Add OwnerReferences for all resource created during deploying vineyard-deployment and support default namespace (#1378) (Ye Cao)
• [afb6b48a]: Add an option to graph loader to measure graph accessing performance (#1377) (Tao He)
• [37f84fd4]: Improve the performance of initDestFidList() when there are many fragments (#1375) (Songqing Zhang)
• [c44c65e5]: Welcome @vegetableysm ✨ (Tao He)
• [fe74ce9b]: Add the missing topologyKey in the schedule workload subcommand of vineyardctl (#1370) (Ye Cao)
• [195dcf95]: Bump github.com/docker/distribution from 2.8.1+incompatible to 2.8.2+incompatible in /k8s (#1371) (dependabot[bot])
• [c2438465]: Warn about duplicated vertices (#1369) (Tao He)
• [eeedd2b1]: Report error when edge src/dst not present in vertices (#1368) (Tao He)
• [bf37a327]: cmake: keep backwards compatibility with cmake<3.18 (#1367) (Tao He)
• [adeacbc3]: Document how to dump the graph (#1366) (Tao He)
• [a64cb46c]: Dump the logs under the namespace which match the wildcard (#1365) (Ye Cao)
• [bd81c72b]: Introduce the kubernetes logs export action and stoat app for debugging the failed e2e test (#1361) (Ye Cao)
• [a99c0bdc]: Print more detail logs when there are unexpected errors in metadata (#1364) (Tao He)
• [470ab448]: Improve the macos CI to build the wheel packages (#1363) (Tao He)