Skip to content

Commit

Permalink
Major refactor
Browse files Browse the repository at this point in the history
This version introduces a major refactor which results in multiple breaking changes. This was done with the intention to make this package the basis for a family of CRDT libraries.

Another motivation was to make this package compatible with [crdt_sync](https://github.com/cachapa/crdt_sync), thereby abstracting the communication protocol and network management for real-time remote synchronization.

Changes:
- Simplified API
- Removed insert and get operations to make package more storage-agnostic
- Made most methods optionally async
- Re-implemented CrdtMap as a zero-effort ephemeral implementation
  • Loading branch information
cachapa committed Sep 10, 2023
1 parent dda170e commit f2b29fc
Show file tree
Hide file tree
Showing 15 changed files with 717 additions and 664 deletions.
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
## 5.0.0
This version introduces a major refactor which results in multiple breaking changes. This was done with the intention to make this package the basis for a family of CRDT libraries.

Another motivation was to make this package compatible with [crdt_sync](https://github.com/cachapa/crdt_sync), thereby abstracting the communication protocol and network management for real-time remote synchronization.

Changes:
- Simplified API
- Removed insert and get operations to make package more storage-agnostic
- Made most methods optionally async
- Reimplemented CrdtMap as a zero-dependency implementation

## 4.0.3
- Update to Dart 3

Expand Down
64 changes: 36 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,56 @@
Dart implementation of Conflict-free Replicated Data Types (CRDTs).

This project is heavily influenced by James Long's talk [CRTDs for Mortals](https://www.dotconferences.com/2019/12/james-long-crdts-for-mortals) and includes a Dart-native implementation of Hybrid Local Clocks (HLC) based on the paper [Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases](https://cse.buffalo.edu/tech-reports/2014-04.pdf).
This project is heavily influenced by James Long's talk [CRTDs for Mortals](https://www.dotconferences.com/2019/12/james-long-crdts-for-mortals) and includes a Dart-native implementation of Hybrid Local Clocks (HLC) based on the paper [Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases](https://cse.buffalo.edu/tech-reports/2014-04.pdf) (pdf).

It has [zero external dependencies](https://github.com/cachapa/crdt/blob/master/pubspec.yaml), so it should run everywhere where Dart runs.
It has [minimal external dependencies](https://github.com/cachapa/crdt/blob/master/pubspec.yaml), so it should run anywhere where Dart runs, which is pretty much everywhere.

See [sql_crdt](https://github.com/cachapa/sql_crdt) for an implementation of CRDTs backed by an SQL database.
The `Crdt` class implements CRDT conflict resolution and serves as a storage-agnostic interface for specific implementations. Als included with this package is `MapCrdt`, an ephemeral implementation using Dart HashMaps.

Other implementations include (so far):
- [hive_crdt](https://github.com/cachapa/hive_crdt), a no-sql implementation using [Hive](https://pub.dev/packages/hive) as persistent storage.
- [sql_crdt](https://github.com/cachapa/sql_crdt), an abstract implementation for using relational databases as a data storage backend.
- [sqlite_crdt](https://github.com/cachapa/sqlite_crdt), an implementation using Sqlite for storage, useful for mobile or small projects.
- [postgres_crdt](https://github.com/cachapa/postgres_crdt), a `sql_crdt` that benefits from PostgreSQL's performance and scalability intended for backend applications.

Moreover, there's [crdt_sync](https://github.com/cachapa/crdt_sync), a turnkey approach for real-time network synchronization of `Crdt` nodes.

## Usage

The `Crdt` class works as a layer on top of a map. The simplest way to experiment is to initialise it with an empty map:
The simplest way to experiment with this package is to use the provided `MapCrdt` implementation:

```dart
import 'package:crdt/crdt.dart';
import 'package:crdt/map_crdt.dart';
void main() {
var crdt = MapCrdt('node_id');
// Insert a record
crdt.put('a', 1);
// Read the record
print('Record: ${crdt.get('a')}');
// Export the CRDT as Json
final json = crdt.toJson();
// Send to remote node
final remoteJson = sendToRemote(json);
// Merge remote CRDT with local
crdt.mergeJson(remoteJson);
// Verify updated record
print('Record after merging: ${crdt.get('a')}');
}
var crdt1 = MapCrdt(['table']);
var crdt2 = MapCrdt(['table']);
print('Inserting 2 records in crdt1…');
crdt1.put('table', 'a', 1);
crdt1.put('table', 'b', 1);
print('crdt1: ${crdt1.get('table')}');
// Mock sending the CRDT to a remote node and getting an updated one back
String sendToRemote(String json) {
final hlc = Hlc.now('another_nodeId');
return '{"a":{"hlc":"$hlc","value":2}}';
print('\nInserting a conflicting record in crdt2…');
crdt2.put('table', 'a', 2);
print('crdt2: ${crdt2.get('table')}');
print('\nMerging crdt2 into crdt1…');
crdt1.merge(crdt2.getChangeset());
print('crdt1: ${crdt1.get('table')}');
}
```

You'll probably want to implement some sort of persistent storage by subclassing the `Crdt` class. An example using [Hive](https://pub.dev/packages/hive) is provided in [hive_crdt](https://github.com/cachapa/hive_crdt).
## Implementations

`crdt` is currently helping build local-first experiences for:

## Example
- [Libra](https://libra-app.eu) a weigh management app with 1M+ installs.
- [tudo](https://github.com/cachapa/tudo) an open-source simple to-do app + backend.

A [simple example](https://github.com/cachapa/crdt/blob/master/example/crdt_example.dart) is provided with this project.
Are you using this package in your project? Let me know!

## Features and bugs

Expand Down
35 changes: 16 additions & 19 deletions example/crdt_example.dart
Original file line number Diff line number Diff line change
@@ -1,25 +1,22 @@
import 'package:crdt/crdt.dart';
import 'package:crdt/src/map_crdt/map_crdt.dart';

void main() {
var crdt = MapCrdt('node_id');
var crdt1 = MapCrdt(['table']);
var crdt2 = MapCrdt(['table']);

// Insert a record
crdt.put('a', 1);
// Read the record
print('Record: ${crdt.get('a')}');
print('Inserting 2 records in crdt1…');
crdt1.put('table', 'a', 1);
crdt1.put('table', 'b', 1);

// Export the CRDT as Json
final json = crdt.toJson();
// Send to remote node
final remoteJson = sendToRemote(json);
// Merge remote CRDT with local
crdt.mergeJson(remoteJson);
// Verify updated record
print('Record after merging: ${crdt.get('a')}');
}
print('crdt1: ${crdt1.getMap('table')}');

print('\nInserting a conflicting record in crdt2…');
crdt2.put('table', 'a', 2);

print('crdt2: ${crdt2.getMap('table')}');

print('\nMerging crdt2 into crdt1…');
crdt1.merge(crdt2.getChangeset());

// Mock sending the CRDT to a remote node and getting an updated one back
String sendToRemote(String json) {
final hlc = Hlc.now('another_nodeId');
return '{"a":{"hlc":"$hlc","value":2}}';
print('crdt1: ${crdt1.getMap('table')}');
}
4 changes: 1 addition & 3 deletions lib/crdt.dart
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
library crdt;

export 'src/crdt.dart';
export 'src/crdt_json.dart';
export 'src/hlc.dart';
export 'src/map_crdt.dart';
export 'src/record.dart';
export 'src/types.dart';
56 changes: 4 additions & 52 deletions lib/map_crdt.dart
Original file line number Diff line number Diff line change
@@ -1,53 +1,5 @@
import 'dart:async';
library map_crdt;

import 'crdt.dart';
import 'hlc.dart';
import 'record.dart';

/// A CRDT backed by a in-memory map.
/// Useful for testing, or for applications which only require temporary datasets.
class MapCrdt<K, V> extends Crdt<K, V> {
final _map = <K, Record<V>>{};
final _controller = StreamController<MapEntry<K, V?>>.broadcast();

@override
final String nodeId;

MapCrdt(this.nodeId, [Map<K, Record<V>> seed = const {}]) {
_map.addAll(seed);
}

@override
bool containsKey(K key) => _map.containsKey(key);

@override
Record<V>? getRecord(K key) => _map[key];

@override
void putRecord(K key, Record<V> value) {
_map[key] = value;
_controller.add(MapEntry(key, value.value));
}

@override
void putRecords(Map<K, Record<V>> recordMap) {
_map.addAll(recordMap);
recordMap
.map((key, value) => MapEntry(key, value.value))
.entries
.forEach(_controller.add);
}

@override
Map<K, Record<V>> recordMap({Hlc? modifiedSince}) =>
Map<K, Record<V>>.from(_map)
..removeWhere((_, record) =>
record.modified.logicalTime < (modifiedSince?.logicalTime ?? 0));

@override
Stream<MapEntry<K, V?>> watch({K? key}) =>
_controller.stream.where((event) => key == null || key == event.key);

@override
void purge() => _map.clear();
}
export 'src/map_crdt/map_crdt.dart';
export 'src/map_crdt/map_crdt_base.dart';
export 'src/map_crdt/record.dart';
Loading

0 comments on commit f2b29fc

Please sign in to comment.