v5.1.0

🚀 Features

• Enhancement 1895: Fully parallelise processing in read_batch (#1950)

🐛 Bugfixes:

• Fix reference counting on python strings (#1999)
• Make snapshot names in run_scenario more unique (#1961)
• Fix Mac ARM Build (#1960)
• Fix delete snapshot so that it doesn't orphan data keys or delete the wrong key (#1973)
• Bugfix 1970: Give helpful error message if and/or/not operators are provided in QueryBuilder operations (#1976)
• Bugfix 127: Improve error message when recursively normalized metastruct is too big (#1981)
  Fix publish result curl fail (#1974)
• Bugfix/1841/maintain empty series names (#1983)
• Bugfix/1937/do not write append ref keys when staging incompletes (#1985)
• Bugfix 1306: Add nonreg test for update failing with 1-nanosecond different timestamps (#1978)
• Test meaningful error message when reading incompletes from non-existent symbol (#1991)
• Bugfix 1655: Disallow / character in mongo lib names (#1992)

Full Changelog: v5.0.0...v5.1.0

The wheels are on PyPI. Below are for debugging:

v5.0.0

⚠️ API Changes

• date_range returned by get_info[_batch] was a datetime.datetime now returns a pandas.Timestamp with nanosecond precision (#1461)
• staged argument removed from write_pickle_batch (#1642)
• Set row_count/rows to None in get_description/get_info and batch versions thereof if symbol is pickled (#1664)
• Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
• Use float64 as the result type for all division operations in the processing pipeline (#1794)
• Make SymbolDescription return type match type hints (#1877)

🚀 Features

• Lazy dataframe implementation (#1703)
• Arbitrary clause ordering (#1860)
• Sort staged data before writing (#1869)
• Make library manager a LRU cache (#1930)
• Preserve nanoseconds and timezones in date range from get info (#1461)
• Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
• Use float64 as the result type for all division operations in the processing pipeline (#1794)

🐛 Fixes

• Improve full build time by 30% by adding PCH (#1764)
• Fix typo in docs flow (#1952)
• Fix entt dep management for conda builds (#1955)
• Run the pull request on the PR sha with the pull_request_target event (#1954)
• Fix the timers macro, configure it with a cmake option (#1914)
• Small fixes for mac arm compilation and fix some tests (#1905)
• Fixes for sort and finalize (#1763)
• Bugfix 1552: Set row_count/rows to None in get_description/get_info and batch versions thereof if symbol is pickled (#1664)
• Bugfix 1641: Remove unused staged arg from write_pickle_batch (#1642)
• Refactor 1749: make processing tests great again (#1758)
• Always use strict weak ordering in std::sort comparators (#1747)
• Use EncodedFieldImpl.blocks() rather than casting member variable _blocks (#1745)
• Fix batch_restore_version failing to use correct next_version_id (#1823)
• Bugfix arcticdb-man 96: Improve error message with pd.Timedelta columns (#1874)
• Bugfix 1865: Make SymbolDescription return type match type hints (#1877)
• Bugfix 1652: Improve error message when object dtype columns contains Timestamps with mixed timezones (#1880)
• Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878)
• Fix library options compatibility across ArcticDB versions (#1862)
• Bugfix 1830: fix resampling with multiindex (#1873)

Uncategorized

• Extend tests for merge sort (#1708)
• Check if object is an instance of QueryBuilder when comparing for equality in QueryBuilder.eq (#1754)
• Add resampling offset (#1743)
• Refactor/1722/remove composite from processing pipeline (#1741)
• Fix Segment use-after-move when replicating to NFS (#1756)
• Throw exception if appending using sort_and_finalize will create unordered index (#1760)
• Improve logging for stress tests (#1759)
• Remove rocksdb (#1761)
• Update BSL table for v4.5.0 (#1768)
• Handle KeyNotFoundException in recurse_index_key (#1766)
• Add Storage API to check for existence of a key matching a predicate (#1762)
• Revert "Handle KeyNotFoundException in recurse_index_key" (#1776)
• Add 1 Billion Row Challenge Notebook (#1774)
• Remove KeyNotFoundException catch in CopyCompressedInterStoreTask (#1778)
• Limit 3 item ref key bypass to only when loading undeleted versions (#1775)
• append docstring improvement: added that append on a new symbol will create it (#1787)
• Parallelize string handling, enable python type handlers, prepare for Arrow (#1698)
• Add interface to pass a cached entry to tombstone all (#1793)
• Fix sort index (#1796)
• Fix windpws CI by adding the string header in decimal.cpp (#1801)
• Remove unused RuntimeConfig feature (#1803)
• Add unimplemented increment counter (#1804)
• Pre arrow refactor (#1805)
• Fix regression in bool pickling (#1810)
• Add API to remove Prometheus metrics (#1806)
• Fix symbol list bug on compaction (#1324) (#1798)
• Allow metrics to be registered twice (#1816)
• Docs 1809: lazy dataframe documentation and example notebook (#1815)
• Bugfix 1818: Fix QueryBuilder equality checks (#1819)
• Support reading from prefixes that include a dot (#1820)
• Fix compilation errors for clang 18 (#1813)
• Add StagedDataFinalizeMethod to docs (#1744)
• Add demo video link to README.md (#1837)
• More sort and finalize fixes (#1799)
• Fix conda build (#1854)
• Fix row slicing with sort_and_finalize throw on column slicing when the column group is larger than the segment column size (#1838)
• Don't override the https value for backwards compatibility (#1840)
• Support the ignore missing key read option in recurse_index_keys (#1844)
• Add flag to clear staged data on failure to both finalize functions (#1856)
• Change test for library emptiness to allow specifying key types to exclude (#1861)
• Fixing clean builds (#1868)
• Extend LMDB instance lifetime (#1879)
• Allow customizing the most useful AWS SDK settings. (#1875)
• Refactor 1833:use entt as ecs (#1834)
• Fix mono CI issue (#1917)
• Fix lib_tool.read_to_keys (#1832)
• Skip lmdb compat tests for versions <= 4.5.0 (#1923)
• Use relative paths for compat tests (#1924)
• ci: Use mamba 2.0 via mamba-org/[email protected] (#1855)
• Remove unneeded step (#1929)
• Adds a type change utility for append data keys in lib_tool (#1932)
• fix introduction help (#1933)
• Rewrite the mkdocs template so the links point to the correct version (#1942)
• Change the pull_request trigger to pull_request_target (#1947)
• Revert pull_request_target back to pull_request (#1949)

New Contributors

• @grusev made their first contribution in #1933
• @maxim-morozov made their first contribution in #1947

Full Changelog: v4.5.1...v5.0.0

---

The wheels are on PyPI. Below are for debugging:

v4.5.1

🐛 Fixes

• Fix bug where libraries created with 4.5.0 are not compatible with older versions by @G-D-Petrov in #1842
• Sort and finalize fixes by @vasil-pashov in #1859
• Extend LMDB instance lifetime (#1879) by @vasil-pashov in #1908

Uncategorized

• Compat tests 451rc by @IvoDD in #1912
• Compat test flakyness fixes 451rc by @IvoDD in #1925
• Bugfix/1785/backport to 4.5.1 by @alexowens90 in #1927

Full Changelog: v4.5.0...v4.5.1

---

The wheels are on PyPI. Below are for debugging:

v4.5.0

🚀 Resampling (#1495)

We are pleased to introduce a first version of resampling into ArcticDB.
Using the QueryBuilder you can easily resample timseries data as part of the read. The following code will resample a symbol into hourly buckets, for example. Please see the QueryBuilder docs for more info.

q = adb.QueryBuilder()
q = q.resample("h").agg({"to_sum": "sum"})
lib.read("symbol", query_builder=q).data

🚀 Features

Remove Composite from processing pipeline (#1722)
Refactor VersionMap to include all combinations of LoadTypes and whether to include_deleted (#1443)
Multiple aggregators per column are now possible in QueryBuilder (#1468)
Add s3 custom ca support (#1338)
Add version_chain ASV benchmarks (#1508)
Support mongodb+srv connection strings (#1513)
Enhancement/515/support metadata in finalize staged data (#1525)
list_snapshots: Add option to skip loading metadata (#1531)
LibraryTool add read_index (#1533)
Make service alive checking more universal (#1607)
Enhancement/1610/method to force compact symbol list (#1624)

🐛 Fixes

Add maybe_unused to fix MSVC compilation error by @vasil-pashov in #1739
Properly hash descriptor in v1 (master) by @willdealtry in #1682
Bugfix 1692: fix batch read with row range no query builder (#1710)
Bugfix 1720: Speedup processing pipeline with lots of columns (#1723)
Fix use after move in op_log.cpp (#1732)
Implement support for very old style symbol list entries. (#1630)
Fix ArcticDB reading streaming data (#1647)
Add validate_index arg to staged data finalization in both V1 and V2 APIs (#1694)
Fix sort_and_finalize_staged_data (#1617)
Bugfix 1609: Allow parallel appends when incomplete index values match last existing index value in the symbol (#1619)
Fix SYMBOL_LIST Segment index out of bounds bug (#1473)
Fix overflow bug on append and update (#1585)
Bugfix/1250/compact incompletes rationalisation (#1467)
Guard against numpy v2 to prevent breakage when it gets released (#1465)
Consistently handle timezones in QueryBuilder filtering (#1483)
Align azure timeout with s3 (#1487)
bugfix/1247: Don't create an lmdb library if the name is invalid. (#1481)
Cleanup arctic client after usage in local_quary_builder in ASV (#1522)
Bugfix/1518/get description date range incorrect after delete data in range (#1523)
Bugfix 1507+1509: dynamic schema different index append fixes (#1529)
Only write ref key once when writing with prune_previous_versions (#1560)
Fix #1148: Show clearer error message for mismatched types in QueryBuilder (#1557)
Bugfix 1615: Fix segfault in benchmark_clause.cpp (#1616)
Bugfix 1612: Raise a meaningful exception type/message when an empty string is provided as a symbol/snapshot name (#1618)
Adapt Raise to use format string as message directly if no args are given (#1606)
Restore mistake in #1315 (#1482)
Update docstring: added section to clarify that an empty df does not delete (#1592)

New Contributors

• @StageTired made their first contribution in #1485

Full Changelog: v4.4.3...v4.5.0

v4.4.3

🐛 Fixes

• Release GIL on update and append (#1559)
• Only write ref key once when writing with prune_previous_versions (#1571)
• Fix overflow bug on append and update (#1585)
• Require numpy < 2 (#1465)

Uncategorized

---

The wheels are on PyPI. Below are for debugging:

v4.4.2

🐛 Fixes

• Fix segmentation fault related to Ray package (by reverting to C++ 17 build) #1542
• Fix get_description date range after using delete_data_in_range #1526
• Fix tail segfault with empty data #1543

---

Please see v4.4.1 release notes for the previous changes

v4.4.1

🐛 Fixes

• Bug fix in Azure Storage, clear to_delete list to allow deletion of more than 256 keys in a batch (#1482)

---

Please see v4.4.0 release notes for the main changes from the last release.

v4.4.0

🚀 Features

• Prevent writing empty types by default (gives compatibility with v1.6.2 readers) in #1440
• Improved resilience to external (out of order) replication in #1355
• Support modifying library options, and introduce enterprise library options in #1457

You can now modify library options on an existing Arctic library:

from arcticdb.options import LibraryOptions

ac: Arctic = ...
lib = ac.create_library("lib")
ac.modify_library_option(lib, ModifiableLibraryOption.ROWS_PER_SEGMENT, int(1e6))

See arcticdb/options.py for a description of the modifiable options.

🐛 Fixes

• Bugfix 902: Cannot filter on nans and nones in string and float columns #1276
• Bugfix: Empty type in #1227
• Bugfix 1334: optimise version ref key access #1345
• Fix version map cache invalidation policies in #1351
• Fix empty column default type in #1378
• Bugfix #1388: Correctly check whether we are in ec2 in #1415
• Bugfix 1423: Raise a meaningful error message when trying to use QueryBuilder with sparse data in #1435
• Bug Fix Windows: Remove lmdb files when delete_library is called in #1437
• Bugfix 1256: reject parallel appends to unsorted data in #1442
• Bugfix 1209: Consistently return metadata from write, append, update, write_metadata, and batch versions thereof in #1444
• Bugfix/935/match pandas behviour when aggregating columns with nans in #1450
• LZ4 decoding empty type: move the segment buffer forward by the compressed data size for empty types in #1463
• Bugfix 1268: swap out xxhash for grouping in #1416
• Bugfix 1260: allow broader range of numeric type promotion with dynamic schema #1426

💾 Storage Exception Normalization

We have made storage-related exceptions uniform across different backend storage platforms, despite the fact that the underlying behaviour varies.

• #447 LMDB Exceptions Normalization in #1285
• #447 Memory Storage Exception Normalization in #1297
• Adds a MockS3Client which can simulate s3 failures in #1281
• #447 S3 Storage Exceptions Normalization in #1304
• Add a MockAzureClient which can simulate azure failures in #1331
• #447 Azure Storage Exceptions Normalization in #1344
• #447 Exception normalization for RocksDB in #1360
• Refactor: Move mongo client errors handling into mongo_storage.cpp before normalization in #1383
• #447 Add a MockMongoClient which can simulate mongo failures in #1395
• #447 Mongo Exceptions Normalization in #1411
• LMDB Exception Normalization with mock client in #1414

Uncategorized

• Prevent writing empty types by default (gives compatibility with v1.6.2 readers) in #1440
• Add a way to enable/disable silencing of errors when deleting a library in #1271
• Feature flag to use WinInet client not WinHttp in #1284
• Abstract an S3ClientWrapper out of details-inl.hpp in #1274
• GitHub Workflows: Make can_merge run for all files to allow any docs changes to be mergeable. in #1292
• Fix benchmarks in #1286
• Reduce the hashes that we use to benchmark against in #1303
• Refactor 1278: Column data dense forward iterator in #1301
• Add contains to Arctic class, to support lib in arctic in #1309
• Update BSL table for v4.3.0 in #1282
• Utility to analyze the size of various key types in a library in #1291
• Do not compile wheel build on EC2 in #1318
• Docs: For top level imports use arcticdb.object instead of using full path to object in #1323
• conda-build: Adaptations for folly in #1320
• Align docstring with the behaviour in #1243
• Added 142 new tests for empty/missing operations in #1319
• Abstract AzureClientWrapper out of azure_storage.cpp in #1315
• Stop using ec2 runners in the conda+linux workflow in #1302
• Adds python tests which simulate s3 storage exceptions in #1330
• S3 local delete failure raising meaningful error in #1329
• Fix dynamic strings append to fixed strings issue in #1346
• Minor improvement on analysis flow in #1290
• Build time improvements in #1263
• build: Disable compilers' extensions in #1335
• maint: Fully specify fmt::format_to in #1333
• Fixed pd_delete_replace + added single tests in #1342
• Use type-deduced functor for all column iterating functions in #1347
• Disable ec2 runners on PR in #1357
• Use 14.39 toolset in #1359
• Removed pytest dependency from arcticdb in #1350
• build: Use C++20 in #1332
• mark test_symbol_list_parallel_stress_with_delete as flaky in #1368
• Docs: Increase CSS max-wdith and build docs from a branch in #1363
• Print error msg in ExponentialBackoff exception in #1365
• Disable missing key warnings when expected in #1379
• Remove pin on civetweb in #1380
• Multiple segments within the same block: storage and library refactoring in #1307
• Not allowing snapshotting tombstoned versions in #1280
• Clarify Intel/AMD build support in #1389
• Fix debug formatting in #1397
• maint: Replace robin_hood with unordered_dense in #1390
• Test benchmarking improvements in #1326
• maint: Remove dependency on some elements of folly in #1370
• Allow different testing dependency version in pipeline in #1410
• Add metadata extraction functions in library_tool in #1375
• Add a global timeout for pytests in #1381
• Set upper bound for supported protobuf version in #1421
• 1 year and 1k stars readme banner in #1425
• read_batch set include_deleted to false by default when reading a version in #1419
• build: Update to fmt 10 in #1427
• Make changes for prometheus metrics in #1418
• maint: Replace use of folly getCurrentThreadId with STL in #1417
• maint: Ignore the diff of #1263 in #1340
• Roll back vcpkg version to fix failing abseil build in #1436
• maint: Remove use of folly/portability/PThread.h in #1447
• maint: Remove use of folly/system/ThreadName.h in #1446
• Better error messaging around pickling in #1451
• Update analysis_workflow.yml in #1455
• Adding update, append and delete asv benchmarks in #1434
• Support generators for metadata vectors again in #1456
• Remove the brotli dep in #1458

---

The wheels are on PyPI. Below are for debugging:

v4.0.4

🚀 Features

• Better backend storage retryable error handling and error message printout #1365

🐛 Fixes

• Stop allowing snapshotting tombstoned versions #1280
• Add a way to enable/disable silencing of errors when deleting a library #1271

---

The wheels are on Pypi. Below are for debugging:

v4.3.1

🐛 Fixes

• Fix regression in round-tripping empty type for dynamic schema (#1313)

---

The wheels are on PyPI. Below are for debugging: