Change file format to version 2

This introduces a counter for the length of tables, so that len() is
constant time instead of linear in the size of the table

CHANGELOG.md

@@ -1,5 +1,20 @@
 # redb - Changelog
 
+## 2.0.0 - 2024-XX-XX
+### Major file format change
+2.0.0 uses a new file format that optimizes `len()` to be constant time. This means that it is not
+backwards compatible with 1.x. To upgrade, consider using a pattern like that in XXX test.
+* `check_integrity()` now returns a `DatabaseError` instead of `StorageError`
+* Refactor table metadata methods into a new `ReadableTableMetadata` trait
+* Rename `RedbKey` to `Key`
+* Rename `RedbValue` to `Value`
+* Remove lifetimes from read-only tables
+* Remove lifetime from `WriteTransaction` and `ReadTransaction`
+* impl `Clone` for `Range`
+* Add `range()` returning a `Range` with the `'static` lifetime to read-only tables
+* Add `get()` returning a range with the `'static` lifetime to read-only multimap tables
+* Add `close()` method to `ReadTransaction`
+
 ## 1.5.0 - 2024-01-15
 * Export `TableStats` type
 * Export `MutInPlaceValue` which allows custom types to support `insert_reserve()`

docs/design.md

@@ -41,16 +41,16 @@ database file.
 | user root page number                                                                          |
 | user root checksum                                                                             |
 | user root checksum (cont.)                                                                     |
+| user root length                                                                               |
 | system root page number                                                                        |
 | system root checksum                                                                           |
 | system root checksum (cont.)                                                                   |
+| system root length                                                                             |
 | freed root page number                                                                         |
 | freed checksum                                                                                 |
 | freed checksum (cont.)                                                                         |
+| freed root length                                                                              |
 | transaction id                                                                                 |
-| padding                                                                                        |
-| padding                                                                                        |
-| padding                                                                                        |
 | slot checksum                                                                                  |
 | slot checksum (cont.)                                                                          |
 ----------------------------------------- Commit slot 1 ------------------------------------------
@@ -117,12 +117,14 @@ This field is only valid when the database does not need recovery. Otherwise it
 * 4 bytes: padding to 64-bit aligned
 * 8 bytes: user root page
 * 16 bytes: user root checksum
+* 8 bytes: user root length
 * 8 bytes: system root page
 * 16 bytes: system root checksum
+* 8 bytes: system root length
 * 8 bytes: freed table root page
 * 16 bytes: freed table root checksum
+* 8 bytes: freed root length
 * 8 bytes: last committed transaction id
-* 24 bytes: padding
 * 16 bytes: slot checksum
 
 `version` the file format version of the database. This is stored in the transaction data, so that it can be atomically
@@ -132,14 +134,20 @@ changed during an upgrade.
 
 `user root checksum` stores the XXH3_128bit checksum of the user root page, which in turn stores the checksum of its child pages.
 
+`user root length` is the number of tables in the user table tree. This field is new in file format v2.
+
 `system root page` is the page number of the root of the system table tree.
 
 `system root checksum` stores the XXH3_128bit checksum of the system root page, which in turn stores the checksum of its child pages.
 
+`system root length` is the number of tables in the system table tree. This field is new in file format v2.
+
 `freed table root page` is the page of the root of the pending free table.
 
 `freed table root checksum` stores the XXH3_128bit checksum of the freed table root page, which in turn stores the checksum of its child pages.
 
+`freed root length` is the length of the freed tree. This field is new in file format v2.
+
 `slot checksum` is the XXH3_128bit checksum of all the preceding fields in the transaction slot.
 
 ### Transaction slot 1 (128 bytes):

src/db.rs

@@ -376,7 +376,7 @@ impl Database {
     ///
     /// Returns `Ok(true)` if the database passed integrity checks; `Ok(false)` if it failed but was repaired,
     /// and `Err(Corrupted)` if the check failed and the file could not be repaired
-    pub fn check_integrity(&mut self) -> Result<bool> {
+    pub fn check_integrity(&mut self) -> Result<bool, DatabaseError> {
         Arc::get_mut(&mut self.mem)
             .unwrap()
             .clear_cache_and_reload()?;

src/error.rs

@@ -1,4 +1,4 @@
-use crate::tree_store::{FILE_FORMAT_VERSION, MAX_VALUE_LENGTH};
+use crate::tree_store::{FILE_FORMAT_VERSION2, MAX_VALUE_LENGTH};
 use crate::{ReadTransaction, TypeName};
 use std::fmt::{Display, Formatter};
 use std::sync::PoisonError;
@@ -222,7 +222,7 @@ impl Display for DatabaseError {
     fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
         match self {
             DatabaseError::UpgradeRequired(actual) => {
-                write!(f, "Manual upgrade required. Expected file format version {FILE_FORMAT_VERSION}, but file is version {actual}")
+                write!(f, "Manual upgrade required. Expected file format version {FILE_FORMAT_VERSION2}, but file is version {actual}")
             }
             DatabaseError::RepairAborted => {
                 write!(f, "Database repair aborted.")
@@ -485,7 +485,7 @@ impl Display for Error {
                 write!(f, "DB corrupted: {msg}")
             }
             Error::UpgradeRequired(actual) => {
-                write!(f, "Manual upgrade required. Expected file format version {FILE_FORMAT_VERSION}, but file is version {actual}")
+                write!(f, "Manual upgrade required. Expected file format version {FILE_FORMAT_VERSION2}, but file is version {actual}")
             }
             Error::ValueTooLarge(len) => {
                 write!(