Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

186081753 - Add topology manager #887

Merged
merged 77 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
9570ec0
wip
sanity Sep 24, 2023
e51bce6
wip
sanity Sep 27, 2023
fa2009c
wip
sanity Sep 27, 2023
e6de98a
wip
sanity Oct 1, 2023
47c305e
Join executor to network ops
iduartgomez Sep 22, 2023
5e69981
Minor fixes to documentation
iduartgomez Sep 26, 2023
25dc2dc
minor text change
sanity Sep 26, 2023
6161c80
minor text change
sanity Sep 26, 2023
036c811
Update dep for the microblogging app
iduartgomez Sep 26, 2023
a050ffb
Update pt_sync.yml
sanity Sep 26, 2023
5c5288e
Update pt_sync.yml
sanity Sep 26, 2023
3420e0f
Remove random bytes delegate request
iduartgomez Sep 26, 2023
581078b
Update crates.io link
iduartgomez Sep 27, 2023
f6e3fec
More documentation renaming
iduartgomez Sep 27, 2023
4262245
Add an email app guide
netsirius Sep 26, 2023
d1a8a3e
Minor name corrections
iduartgomez Sep 27, 2023
eebcdc7
wip: fix app working
iduartgomez Sep 27, 2023
5a91db2
Messaging app runs
iduartgomez Sep 28, 2023
8072ca6
Package wasm with debug symbols if set
iduartgomez Sep 28, 2023
7aa8868
Fix issues with packaging webapps
iduartgomez Sep 29, 2023
d49b8c7
Update tutorial
iduartgomez Sep 29, 2023
6b042f9
top: add topology manager
sanity Oct 1, 2023
8720017
top: testing simulation
sanity Oct 6, 2023
188be57
Add submodules to audit action
iduartgomez Oct 2, 2023
3e8fa06
Bump styfle/cancel-workflow-action from 0.11.0 to 0.12.0
dependabot[bot] Oct 3, 2023
41528d4
#186148680 - Layer1 contract abstractions (#861)
iduartgomez Oct 10, 2023
4c02d43
fix: upgrade bootstrap from 5.3.1 to 5.3.2
snyk-bot Oct 6, 2023
ec08c92
Update stdlib ver (#866)
iduartgomez Oct 10, 2023
ec01e01
Propagate router estimation errors (#867)
sanity Oct 11, 2023
c6200c5
top: clean up simulator
sanity Oct 11, 2023
c94e203
Connect router and feed live events
iduartgomez Oct 11, 2023
7493178
Log route events and reload router periodically
iduartgomez Oct 12, 2023
3b8a0de
Use router to route requests.
iduartgomez Oct 12, 2023
e7a1194
Truncate record when maxed
iduartgomez Oct 12, 2023
ebe304e
Don't load old records
iduartgomez Oct 13, 2023
4d1202d
top: sim testing
sanity Oct 18, 2023
3122187
Bump up freenet version to "0.0.6" for freenet-token-generator
KristijanZic Oct 15, 2023
5a31945
Properly concantenate compiler features
iduartgomez Oct 16, 2023
d2b2c1f
Rename fdev strings from 'Locutus' to 'Freenet' (#871)
KristijanZic Oct 16, 2023
3fc1c8b
Rename fdev variable prefixes from "locutus" to "freenet" (#873)
KristijanZic Oct 18, 2023
c6ca732
top: add connection connection evaluator
sanity Oct 19, 2023
c9c9d76
Fix API breaking changes from libp2p (#874)
iduartgomez Oct 19, 2023
daa4351
Fix API breaking changes from libp2p (#874)
iduartgomez Oct 19, 2023
cc40eba
Generate documentation per-release
sanity Oct 21, 2023
f9958b2
Create docs-retro.yml
sanity Oct 21, 2023
3a82b4a
Update docs-retro.yml
sanity Oct 21, 2023
f4aa11f
Update docs.yml
sanity Oct 21, 2023
93e7787
Update docs.yml
sanity Oct 21, 2023
a19dce0
Update README.md
sanity Oct 21, 2023
d059188
Update docs.yml
sanity Oct 21, 2023
9a2e146
Bump actions/checkout from 2 to 4 (#875)
dependabot[bot] Oct 24, 2023
1076669
Bump actions/cache from 2 to 3 (#876)
dependabot[bot] Oct 24, 2023
bb0dd3f
Make sim network compile
iduartgomez Oct 13, 2023
e6590d6
Add mock executor impl
iduartgomez Oct 15, 2023
aea5907
Get the `get` sim network tests to pass
iduartgomez Oct 16, 2023
a0574f3
Multiple fixes around join ring op retries
iduartgomez Oct 17, 2023
184dcdd
Actually return back the populated connection list
iduartgomez Oct 17, 2023
da8a1d7
Fixes to event log and sim connectivity report
iduartgomez Oct 18, 2023
78d614e
Cleanup a bunch of unnecessary code and todos
iduartgomez Oct 19, 2023
fe4e8f3
Add rand bytes func
iduartgomez Oct 19, 2023
5405eff
Rename join ring to connect
iduartgomez Oct 19, 2023
e517445
Fix some issues with put op + remove some todos
iduartgomez Oct 20, 2023
d12c01b
Fix issue of not caching contract after get
iduartgomez Oct 20, 2023
b566f52
Pass subscribe op test
iduartgomez Oct 23, 2023
b254243
Handle concurrent same tx messages gracefully
iduartgomez Oct 24, 2023
f0c5117
Fix CI issues
iduartgomez Oct 24, 2023
891ec7b
Add garbage op cleanup background task
iduartgomez Oct 24, 2023
f88fc06
Update deps
iduartgomez Oct 24, 2023
c8cc46d
tm: sliding windows
sanity Oct 25, 2023
130e815
tm: RequestDensityTracker complete and tested
sanity Nov 1, 2023
c3042af
tm: unit tests + logging
sanity Nov 1, 2023
3eb859c
Improving Developer Onboarding Documentation (#883)
kernelkind Nov 2, 2023
7dba1d2
tm: topology manager tests pass
sanity Nov 5, 2023
97d2649
tm: improve docs
sanity Nov 5, 2023
fe8b0ca
tm: cleanup clippy warnings
iduartgomez Nov 6, 2023
e72341f
Merge remote-tracking branch 'origin' into 186081753-add-topology-man…
iduartgomez Nov 6, 2023
9b71bda
tm: cargo fmt
iduartgomez Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
649 changes: 410 additions & 239 deletions Cargo.lock

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions crates/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ tracing = "0.1"
arbitrary = { version = "1", features = ["derive"] }
itertools = "0.11"
pico-args = "0.5"
statrs = "0.16.0"
freenet-stdlib = { workspace = true, features = ["testing", "net"] }

[features]
Expand Down
1 change: 1 addition & 0 deletions crates/core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ mod router;
mod runtime;
#[cfg(feature = "websocket")]
pub mod server;
mod topology;
pub mod util;

type DynError = Box<dyn std::error::Error + Send + Sync + 'static>;
Expand Down
35 changes: 32 additions & 3 deletions crates/core/src/ring.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@
//! - next node
//! - final location

use std::hash::Hash;
use std::{
collections::BTreeMap,
convert::TryFrom,
fmt::Display,
hash::Hasher,
ops::Add,
sync::{
atomic::{AtomicU64, AtomicUsize, Ordering::SeqCst},
Arc,
Expand Down Expand Up @@ -446,6 +448,11 @@ impl Location {
Location(location)
}

/// Returns a new location rounded to ensure it is between 0.0 and 1.0
pub fn new_rounded(location: f64) -> Self {
Self::new(location.rem_euclid(1.0))
}

/// Returns a new random location.
pub fn random() -> Self {
use rand::prelude::*;
Expand All @@ -462,6 +469,10 @@ impl Location {
Distance::new(1.0f64 - d)
}
}

pub fn as_f64(&self) -> f64 {
self.0
}
}

/// Ensure at compile time locations can only be constructed from well formed contract keys
Expand Down Expand Up @@ -535,23 +546,41 @@ impl Distance {
pub fn new(value: f64) -> Self {
debug_assert!(!value.is_nan(), "Distance cannot be NaN");
debug_assert!(
(0.0..=0.5).contains(&value),
"Distance must be in the range [0, 0.5]"
(0.0..=1.0).contains(&value),
"Distance must be in the range [0, 1.0]"
);
Distance(value)
if value <= 0.5 {
Distance(value)
} else {
Distance(1.0 - value)
}
}

pub fn as_f64(&self) -> f64 {
self.0
}
}

impl Add for Distance {
type Output = Self;

fn add(self, rhs: Self) -> Self::Output {
let d = self.0 + rhs.0;
if d > 0.5 {
Distance::new(1.0 - d)
} else {
Distance::new(d)
}
}
}

impl PartialEq for Distance {
fn eq(&self, other: &Self) -> bool {
(self.0 - other.0).abs() < f64::EPSILON
}
}

#[allow(clippy::incorrect_partial_ord_impl_on_ord_type)]
impl PartialOrd for Distance {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
Expand Down
63 changes: 63 additions & 0 deletions crates/core/src/topology/connection_evaluator/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
use std::collections::VecDeque;
use std::time::{Duration, Instant};

/// `ConnectionEvaluator` is used to evaluate connection scores within a specified time window.
///
/// The evaluator records scores and determines whether a given score is better (higher) than
/// any other scores within a predefined time window. A score is considered better if it's higher
/// than all other scores in the time window, or if no scores were recorded within the window's
/// duration.
///
/// In the Freenet context, this will be used to titrate the rate of new connection requests accepted
/// by a node. The node will only accept a new connection if the score of the connection is better
/// than all other scores within the time window.
pub(crate) struct ConnectionEvaluator {
scores: VecDeque<(Instant, f64)>,
window_duration: Duration,
}

impl ConnectionEvaluator {
pub fn new(window_duration: Duration) -> Self {
ConnectionEvaluator {
scores: VecDeque::new(),
window_duration,
}
}

pub fn record_only(&mut self, score: f64) {
self.record_only_with_current_time(score, Instant::now());
}

pub fn record_only_with_current_time(&mut self, score: f64, current_time: Instant) {
self.remove_outdated_scores(current_time);
self.scores.push_back((current_time, score));
}

pub fn record_and_eval(&mut self, score: f64) -> bool {
self.record_and_eval_with_current_time(score, Instant::now())
}

pub fn record_and_eval_with_current_time(&mut self, score: f64, current_time: Instant) -> bool {
self.remove_outdated_scores(current_time);

let is_better = self.scores.is_empty() || self.scores.iter().all(|&(_, s)| score > s);

// Important to add new score *after* checking if it's better than all other scores
self.record_only_with_current_time(score, current_time);

is_better
}

fn remove_outdated_scores(&mut self, current_time: Instant) {
while let Some(&(time, _)) = self.scores.front() {
if current_time.duration_since(time) > self.window_duration {
self.scores.pop_front();
} else {
break;
}
}
}
}

#[cfg(test)]
mod tests;
84 changes: 84 additions & 0 deletions crates/core/src/topology/connection_evaluator/tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
use super::*;

#[test]
fn test_record_first_score() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));
let current_time = Instant::now();
assert!(evaluator.record_and_eval_with_current_time(5.0, current_time));
// Assert evaluator.scores contains the new score
assert_eq!(evaluator.scores.len(), 1);
assert_eq!(evaluator.scores[0].1, 5.0);
assert_eq!(evaluator.scores[0].0, current_time);
}

#[test]
fn test_not_best_in_time_window() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));

let start_time = Instant::now();
evaluator.record_and_eval_with_current_time(5.0, start_time);
assert!(!evaluator.record_and_eval_with_current_time(4.0, start_time + Duration::from_secs(5)),);
}

#[test]
fn test_best_in_time_window() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));

let start_time = Instant::now();
evaluator.record_and_eval_with_current_time(5.0, start_time);
assert!(evaluator.record_and_eval_with_current_time(4.0, start_time + Duration::from_secs(11)),);
}

#[test]
fn test_remove_outdated_scores() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));

let start_time = Instant::now();
evaluator.record_and_eval_with_current_time(5.0, start_time);
evaluator.record_and_eval_with_current_time(6.0, start_time + Duration::from_secs(5));
evaluator.record_and_eval_with_current_time(4.5, start_time + Duration::from_secs(11));
assert_eq!(evaluator.scores.len(), 2);
}

#[test]
fn test_empty_window_duration() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(0));
let current_time = Instant::now();
assert!(evaluator.record_and_eval_with_current_time(5.0, current_time));
assert!(!evaluator.record_and_eval_with_current_time(4.0, current_time));
}

#[test]
fn test_multiple_scores_same_timestamp() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));
let current_time = Instant::now();
evaluator.record_only_with_current_time(5.0, current_time);
evaluator.record_only_with_current_time(6.0, current_time);
assert_eq!(evaluator.scores.len(), 2);
assert!(
!evaluator.record_and_eval_with_current_time(4.0, current_time + Duration::from_secs(5)),
);
}

#[test]
fn test_negative_scores() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));
let start_time = Instant::now();
assert!(evaluator.record_and_eval_with_current_time(-5.0, start_time),);
assert!(evaluator.record_and_eval_with_current_time(-4.0, start_time + Duration::from_secs(5)),);
assert!(
!evaluator.record_and_eval_with_current_time(-6.0, start_time + Duration::from_secs(5)),
);
}

#[test]
fn test_large_number_of_scores() {
let mut evaluator = ConnectionEvaluator::new(Duration::from_secs(10));
let start_time = Instant::now();
for i in 0..1000 {
evaluator.record_only_with_current_time(i as f64, start_time + Duration::from_secs(i));
}
assert!(
evaluator.record_and_eval_with_current_time(1000.0, start_time + Duration::from_secs(1001)),
);
}
Loading
Loading