Skip to content

Commit

Permalink
Merge pull request #1806 from AntelopeIO/merge-main-10-23-23
Browse files Browse the repository at this point in the history 
IF: Merge main into hotstuff_integration (10/23/2023)
  • Loading branch information
@linh2931
linh2931 authored Oct 23, 2023
2 parents be67f89 + 6e597e3 commit 6475d08
Show file tree
Hide file tree
Showing 27 changed files with 350 additions and 427 deletions.
143 changes: 70 additions & 73 deletions docs/01_nodeos/03_plugins/producer_plugin/10_block-producing-explained.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,87 +4,84 @@ content_title: Block Production Explained

For simplicity of the explanation let's consider the following notations:

m = max_block_cpu_usage
* `r` = `producer_repetitions = 12` (hard-coded value)
* `m` = `max_block_cpu_usage` (on-chain consensus value)
* `u` = `max_block_net_usage` (on-chain consensus value)
* `t` = `block-time`
* `e` = `produce-block-offset-ms` (nodeos configuration)
* `w` = `block-time-interval = 500ms` (hard-coded value)
* `a` = `produce-block-early-amount = w - (w - (e / r)) = e / r ms` (how much to release each block of round early by)
* `l` = `produce-block-time = t - a`
* `p` = `produce block time window = w - a` (amount of wall clock time to produce a block)
* `c` = `billed_cpu_in_block = minimum(m, w - a)`
* `n` = `network tcp/ip latency`
* `h` = `block header validation time ms`

Peer validation for similar hardware/version/config will be <= `m`

**Let's consider the example of the following two BPs and their network topology as depicted in the below diagram**

t = block-time

e = last-block-cpu-effort-percent

w = block_time_interval = 500ms

a = produce-block-early-amount = (w - w*e/100) ms

p = produce-block-time; p = t - a

c = billed_cpu_in_block = minimum(m, w - a)

n = network tcp/ip latency

peer validation for similar hardware/eosio-version/config will be <= m

**Let's consider for exemplification the following four BPs and their network topology as depicted in below diagram**


```dot-svg
#p2p_local_chain_prunning.dot - local chain prunning
#
#notes: * to see image copy/paste to https://dreampuf.github.io/GraphvizOnline
# * image will be rendered by gatsby-remark-graphviz plugin in eosio docs.
digraph {
newrank=true #allows ranks inside subgraphs (important!)
compound=true #allows edges connecting nodes with subgraphs
graph [rankdir=LR]
node [style=filled, fillcolor=lightgray, shape=square, fixedsize=true, width=.55, fontsize=10]
edge [dir=both, arrowsize=.6, weight=100]
splines=false
subgraph cluster_chain {
label="Block Producers Peers"; labelloc="b"
graph [color=invis]
b0 [label="...", color=invis, style=""]
b1 [label="BP-A"]; b2 [label="BP-A\nPeer"]; b3 [label="BP-B\nPeer"]; b4 [label="BP-B"]
b5 [label="...", color=invis, style=""]
b0 -> b1 -> b2 -> b3 -> b4 -> b5
} //cluster_chain
} //digraph
```
+------+ +------+ +------+ +------+
-->| BP-A |---->| BP-A |------>| BP-B |---->| BP-B |
+------+ | Peer | | Peer | +------+
+------+ +------+
```

`BP-A` will send block at `p` and,

`BP-B` needs block at time `t` or otherwise will drop it.
`BP-A` will send block at `l` and, `BP-B` needs block at time `t` or otherwise will drop it.

If `BP-A`is producing 12 blocks as follows `b(lock) at t(ime) 1`, `bt 1.5`, `bt 2`, `bt 2.5`, `bt 3`, `bt 3.5`, `bt 4`, `bt 4.5`, `bt 5`, `bt 5.5`, `bt 6`, `bt 6.5` then `BP-B` needs `bt 6.5` by time `6.5` so it has `.5` to produce `bt 7`.

Please notice that the time of `bt 7` minus `.5` equals the time of `bt 6.5` therefore time `t` is the last block time of `BP-A` and when `BP-B` needs to start its first block.

## Example 1
`BP-A` has 50% e, m = 200ms, c = 200ms, n = 0ms, a = 250ms:
`BP-A` sends at (t-250ms) <-> `BP-A-Peer` processes for 200ms and sends at (t - 50ms) <-> `BP-B-Peer` processes for 200ms and sends at (t + 150ms) <-> arrive at `BP-B` 150ms too late.

## Example 2
`BP-A` has 40% e and m = 200ms, c = 200ms, n = 0ms, a = 300ms:
(t-300ms) <-> (+200ms) <-> (+200ms) <-> arrive at `BP-B` 100ms too late.

## Example 3
`BP-A` has 30% e and m = 200ms, c = 150ms, n = 0ms, a = 350ms:
(t-350ms) <-> (+150ms) <-> (+150ms) <-> arrive at `BP-B` with 50ms to spare.

## Example 4
`BP-A` has 25% e and m = 200ms, c = 125ms, n = 0ms, a = 375ms:
(t-375ms) <-> (+125ms) <-> (+125ms) <-> arrive at `BP-B` with 125ms to spare.

## Example 5
`BP-A` has 10% e and m = 200ms, c = 50ms, n = 0ms, a = 450ms:
(t-450ms) <-> (+50ms) <-> (+50ms) <-> arrive at `BP-B` with 350ms to spare.

## Example 6
`BP-A` has 10% e and m = 200ms, c = 50ms, n = 15ms, a = 450ms:
(t-450ms) <- +15ms -> (+50ms) <- +15ms -> (+50ms) <- +15ms -> `BP-B` <-> arrive with 305ms to spare.
A block is produced and sent when either it reaches `m` or `u` or `p`.

Starting in Leap 4.0, blocks are propagated after block header validation. This means instead of `BP-A Peer` & `BP-B Peer` taking `m` time to validate and forward a block it only takes a small number of milliseconds to verify the block header and then forward the block.

Starting in Leap 5.0, blocks in a round are started immediately after the completion of the previous block. Before 5.0, blocks were always started on `w` intervals and a node would "sleep" between blocks if needed. In 5.0, the "sleeps" are all moved to the end of the block production round.

## Example 1: block arrives 110ms early
* Assuming zero network latency between all nodes.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 120, n = 0ms, h = 5ms, a = 10ms
* `BP-A` sends b1 at `t1-10ms` => `BP-A-Peer` processes `h=5ms`, sends at `t-5ms` => `BP-B-Peer` processes `h=5ms`, sends at `t-0ms` => arrives at `BP-B` at `t`.
* `BP-A` starts b2 at `t1-10ms`, sends b2 at `t2-20ms` => `BP-A-Peer` processes `h=5ms`, sends at `t2-15ms` => `BP-B-Peer` processes `h=5ms`, sends at `t2-10ms` => arrives at `BP-B` at `t2-10ms`.
* `BP-A` starts b3 at `t2-20ms`, ...
* `BP-A` starts b12 at `t11-110ms`, sends b12 at `t12-120ms` => `BP-A-Peer` processes `h=5ms`, sends at `t12-115ms` => `BP-B-Peer` processes `h=5ms`, sends at `t12-110ms` => arrives at `BP-B` at `t12-110ms`

## Example 2: block arrives 80ms early
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 150ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 240, n = 0ms/150ms, h = 5ms, a = 20ms
* `BP-A` sends b1 at `t1-20ms` => `BP-A-Peer` processes `h=5ms`, sends at `t-15ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t+140ms` => arrives at `BP-B` at `t+140ms`.
* `BP-A` starts b2 at `t1-20ms`, sends b2 at `t2-40ms` => `BP-A-Peer` processes `h=5ms`, sends at `t2-35ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t2+120ms` => arrives at `BP-B` at `t2+120ms`.
* `BP-A` starts b3 at `t2-40ms`, ...
* `BP-A` starts b12 at `t11-220ms`, sends b12 at `t12-240ms` => `BP-A-Peer` processes `h=5ms`, sends at `t12-235ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t12-80ms` => arrives at `BP-B` at `t12-80ms`

## Example 3: block arrives 16ms late and is dropped
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 200ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 204, n = 0ms/200ms, h = 10ms, a = 17ms
* `BP-A` sends b1 at `t1-17ms` => `BP-A-Peer` processes `h=10ms`, sends at `t-7ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t+203ms` => arrives at `BP-B` at `t+203ms`.
* `BP-A` starts b2 at `t1-17ms`, sends b2 at `t2-34ms` => `BP-A-Peer` processes `h=10ms`, sends at `t2-24ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t2+186ms` => arrives at `BP-B` at `t2+186ms`.
* `BP-A` starts b3 at `t2-34ms`, ...
* `BP-A` starts b12 at `t11-187ms`, sends b12 at `t12-204ms` => `BP-A-Peer` processes `h=10ms`, sends at `t12-194ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t12+16ms` => arrives at `BP-B` at `t12+16ms`

## Example 4: full blocks are produced early
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 200ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assume all blocks are full as there are enough queued up unapplied transactions ready to fill all blocks.
* Assume a block can be produced with 200ms worth of transactions in 225ms worth of time. There is overhead for producing the block.
* `BP-A` has e = 120, m = 200ms, n = 0ms/200ms, h = 10ms, a = 10ms
* `BP-A` sends b1 at `t1-275s` => `BP-A-Peer` processes `h=10ms`, sends at `t-265ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t-55ms` => arrives at `BP-B` at `t-55ms`.
* `BP-A` starts b2 at `t1-275ms`, sends b2 at `t2-550ms (t1-50ms)` => `BP-A-Peer` processes `h=10ms`, sends at `t2-540ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t2-330ms` => arrives at `BP-B` at `t2-330ms`.
* `BP-A` starts b3 at `t2-550ms`, ...
* `BP-A` starts b12 at `t11-3025ms`, sends b12 at `t12-3300ms` => `BP-A-Peer` processes `h=10ms`, sends at `t12-3290ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t12-3080ms` => arrives at `BP-B` at `t12-3080ms`

## Example 7
Example world-wide network:`BP-A`has 10% e and m = 200ms, c = 50ms, n = 15ms/250ms, a = 450ms:
(t-450ms) <- +15ms -> (+50ms) <- +250ms -> (+50ms) <- +15ms -> `BP-B` <-> arrive with 70ms to spare.

Running wasm-runtime=eos-vm-jit eos-vm-oc-enable on relay node will reduce the validation time.
17 changes: 3 additions & 14 deletions docs/01_nodeos/03_plugins/producer_plugin/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,20 +72,9 @@ Config Options for eosio::producer_plugin:
can extend during low usage (only
enforced subjectively; use 1000 to not
enforce any limit)
--produce-time-offset-us arg (=0) Offset of non last block producing time
in microseconds. Valid range 0 ..
-block_time_interval.
--last-block-time-offset-us arg (=-200000)
Offset of last block producing time in
microseconds. Valid range 0 ..
-block_time_interval.
--cpu-effort-percent arg (=80) Percentage of cpu block production time
used to produce block. Whole number
percentages, e.g. 80 for 80%
--last-block-cpu-effort-percent arg (=80)
Percentage of cpu block production time
used to produce last block. Whole
number percentages, e.g. 80 for 80%
--produce-block-offset-ms arg (=450) The minimum time to reserve at the end
of a production round for blocks to
propagate to the next block producer.
--max-block-cpu-usage-threshold-us arg (=5000)
Threshold of CPU block production to
consider block full; when within
Expand Down
2 changes: 1 addition & 1 deletion libraries/chain/include/eosio/chain/config.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ const static uint32_t default_max_inline_action_size = 512 * 102
const static uint16_t default_max_inline_action_depth = 4;
const static uint16_t default_max_auth_depth = 6;
const static uint32_t default_sig_cpu_bill_pct = 50 * percent_1; // billable percentage of signature recovery
const static uint32_t default_block_cpu_effort_pct = 90 * percent_1; // percentage of block time used for producing block
const static uint32_t default_produce_block_offset_ms = 450;
const static uint16_t default_controller_thread_pool_size = 2;
const static uint32_t default_max_variable_signature_length = 16384u;
const static uint32_t default_max_action_return_value_size = 256;
Expand Down
30 changes: 23 additions & 7 deletions libraries/chain/webassembly/runtimes/eos-vm-oc/code_cache.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -226,13 +226,13 @@ const code_descriptor* const code_cache_sync::get_descriptor_for_code_sync(const

code_cache_base::code_cache_base(const std::filesystem::path& data_dir, const eosvmoc::config& eosvmoc_config, const chainbase::database& db) :
_db(db),
_cache_file_path(data_dir/"code_cache.bin")
{
_cache_file_path(data_dir/"code_cache.bin") {
static_assert(sizeof(allocator_t) <= header_offset, "header offset intersects with allocator");

std::filesystem::create_directories(data_dir);

if(!std::filesystem::exists(_cache_file_path)) {
bool created_file = false;
auto create_code_cache_file = [&] {
EOS_ASSERT(eosvmoc_config.cache_size >= allocator_t::get_min_size(total_header_size), database_exception, "configured code cache size is too small");
std::ofstream ofs(_cache_file_path.generic_string(), std::ofstream::trunc);
EOS_ASSERT(ofs.good(), database_exception, "unable to create EOS VM Optimized Compiler code cache");
Expand All @@ -241,19 +241,35 @@ code_cache_base::code_cache_base(const std::filesystem::path& data_dir, const eo
bip::mapped_region creation_region(creation_mapping, bip::read_write);
new (creation_region.get_address()) allocator_t(eosvmoc_config.cache_size, total_header_size);
new ((char*)creation_region.get_address() + header_offset) code_cache_header;
}
created_file = true;
};

code_cache_header cache_header;
{
auto check_code_cache = [&] {
char header_buff[total_header_size];
std::ifstream hs(_cache_file_path.generic_string(), std::ifstream::binary);
hs.read(header_buff, sizeof(header_buff));
EOS_ASSERT(!hs.fail(), bad_database_version_exception, "failed to read code cache header");
memcpy((char*)&cache_header, header_buff + header_offset, sizeof(cache_header));

EOS_ASSERT(cache_header.id == header_id, bad_database_version_exception, "existing EOS VM OC code cache not compatible with this version");
EOS_ASSERT(!cache_header.dirty, database_exception, "code cache is dirty");
};

if (!std::filesystem::exists(_cache_file_path)) {
create_code_cache_file();
}

EOS_ASSERT(cache_header.id == header_id, bad_database_version_exception, "existing EOS VM OC code cache not compatible with this version");
EOS_ASSERT(!cache_header.dirty, database_exception, "code cache is dirty");
try {
check_code_cache();
} catch (const fc::exception&) {
if (created_file)
throw;

ilog("EOS VM optimized Compiler code cache corrupt, recreating");
create_code_cache_file();
check_code_cache();
}

set_on_disk_region_dirty(true);

Expand Down
2 changes: 1 addition & 1 deletion libraries/chainbase
Submodule chainbase updated 1 files
+4 −2 include/chainbase/undo_index.hpp
53 changes: 14 additions & 39 deletions libraries/state_history/include/eosio/state_history/log.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,10 @@ struct state_history_log_header {
chain::block_id_type block_id = {};
uint64_t payload_size = 0;
};
static const int state_history_log_header_serial_size = sizeof(state_history_log_header::magic) +
sizeof(state_history_log_header::block_id) +
sizeof(state_history_log_header::payload_size);
static constexpr int state_history_log_header_serial_size = sizeof(state_history_log_header::magic) +
sizeof(state_history_log_header::block_id) +
sizeof(state_history_log_header::payload_size);
static_assert(sizeof(state_history_log_header) == state_history_log_header_serial_size);

namespace state_history {
struct prune_config {
Expand Down Expand Up @@ -323,7 +324,7 @@ class state_history_log {
catalog.open(log_dir, conf.retained_dir, conf.archive_dir, name);
catalog.max_retained_files = conf.max_retained_files;
if (_end_block == 0) {
_begin_block = _end_block = catalog.last_block_num() +1;
_index_begin_block = _begin_block = _end_block = catalog.last_block_num() +1;
}
}
}, _config);
Expand Down Expand Up @@ -539,6 +540,7 @@ class state_history_log {
"wrote payload with incorrect size to ${name}.log", ("name", name));
fc::raw::pack(log, pos);

index.seek_end(0);
fc::raw::pack(index, pos);
if (_begin_block == _end_block)
_index_begin_block = _begin_block = block_num;
Expand Down Expand Up @@ -576,10 +578,14 @@ class state_history_log {
if (block_num >= _begin_block && block_num < _end_block) {
state_history_log_header header;
get_entry(block_num, header);
EOS_ASSERT(chain::block_header::num_from_id(header.block_id) == block_num, chain::plugin_exception,
"header id does not match requested ${a} != ${b}", ("a", chain::block_header::num_from_id(header.block_id))("b", block_num));
return header.block_id;
}
return {};
}
EOS_ASSERT(chain::block_header::num_from_id(*result) == block_num, chain::plugin_exception,
"catalog id does not match requested ${a} != ${b}", ("a", chain::block_header::num_from_id(*result))("b", block_num));
return result;
}

Expand Down Expand Up @@ -894,47 +900,16 @@ class state_history_log {
}

void split_log() {

std::filesystem::path log_file_path = log.get_file_path();
std::filesystem::path index_file_path = index.get_file_path();

fc::datastream<fc::cfile> new_log_file;
fc::datastream<fc::cfile> new_index_file;

std::filesystem::path tmp_log_file_path = log_file_path;
tmp_log_file_path.replace_extension("log.tmp");
std::filesystem::path tmp_index_file_path = index_file_path;
tmp_index_file_path.replace_extension("index.tmp");

new_log_file.set_file_path(tmp_log_file_path);
new_index_file.set_file_path(tmp_index_file_path);

try {
new_log_file.open(fc::cfile::truncate_rw_mode);
new_index_file.open(fc::cfile::truncate_rw_mode);

} catch (...) {
wlog("Unable to open new state history log or index file for writing during log spliting, "
"continue writing to existing block log file\n");
return;
}

index.close();
log.close();

catalog.add(_begin_block, _end_block - 1, log.get_file_path().parent_path(), name);

_begin_block = _end_block;
_index_begin_block = _begin_block = _end_block;

using std::swap;
swap(new_log_file, log);
swap(new_index_file, index);

std::filesystem::rename(tmp_log_file_path, log_file_path);
std::filesystem::rename(tmp_index_file_path, index_file_path);

log.set_file_path(log_file_path);
index.set_file_path(index_file_path);
log.open(fc::cfile::truncate_rw_mode);
log.seek_end(0);
index.open(fc::cfile::truncate_rw_mode);
}
}; // state_history_log

Expand Down
10 changes: 1 addition & 9 deletions plugins/chain_plugin/chain_plugin.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -463,14 +463,6 @@ void clear_directory_contents( const std::filesystem::path& p ) {
}
}

void clear_chainbase_files( const std::filesystem::path& p ) {
if( !std::filesystem::is_directory( p ) )
return;

std::filesystem::remove( p / "shared_memory.bin" );
std::filesystem::remove( p / "shared_memory.meta" );
}

namespace {
// This can be removed when versions of eosio that support reversible chainbase state file no longer supported.
void upgrade_from_reversible_to_fork_db(chain_plugin_impl* my) {
Expand Down Expand Up @@ -762,7 +754,7 @@ void chain_plugin_impl::plugin_initialize(const variables_map& options) {
ilog( "Replay requested: deleting state database" );
if( options.at( "truncate-at-block" ).as<uint32_t>() > 0 )
wlog( "The --truncate-at-block option does not work for a regular replay of the blockchain." );
clear_chainbase_files( chain_config->state_dir );
clear_directory_contents( chain_config->state_dir );
} else if( options.at( "truncate-at-block" ).as<uint32_t>() > 0 ) {
wlog( "The --truncate-at-block option can only be used with --hard-replay-blockchain." );
}
Expand Down
Loading

0 comments on commit 6475d08

Please sign in to comment.