Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/release/5.0' into GH-1784-cpu-ef…
Browse files Browse the repository at this point in the history 
…fort-main
  • Loading branch information
@heifner
heifner committed Oct 21, 2023
2 parents d7caa21 + 03a16be commit d76783c
Show file tree
Hide file tree
Showing 10 changed files with 140 additions and 147 deletions.
143 changes: 70 additions & 73 deletions docs/01_nodeos/03_plugins/producer_plugin/10_block-producing-explained.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,87 +4,84 @@ content_title: Block Production Explained

For simplicity of the explanation let's consider the following notations:

m = max_block_cpu_usage
* `r` = `producer_repetitions = 12` (hard-coded value)
* `m` = `max_block_cpu_usage` (on-chain consensus value)
* `u` = `max_block_net_usage` (on-chain consensus value)
* `t` = `block-time`
* `e` = `produce-block-offset-ms` (nodeos configuration)
* `w` = `block-time-interval = 500ms` (hard-coded value)
* `a` = `produce-block-early-amount = w - (w - (e / r)) = e / r ms` (how much to release each block of round early by)
* `l` = `produce-block-time = t - a`
* `p` = `produce block time window = w - a` (amount of wall clock time to produce a block)
* `c` = `billed_cpu_in_block = minimum(m, w - a)`
* `n` = `network tcp/ip latency`
* `h` = `block header validation time ms`

Peer validation for similar hardware/version/config will be <= `m`

**Let's consider the example of the following two BPs and their network topology as depicted in the below diagram**

t = block-time

e = last-block-cpu-effort-percent

w = block_time_interval = 500ms

a = produce-block-early-amount = (w - w*e/100) ms

p = produce-block-time; p = t - a

c = billed_cpu_in_block = minimum(m, w - a)

n = network tcp/ip latency

peer validation for similar hardware/eosio-version/config will be <= m

**Let's consider for exemplification the following four BPs and their network topology as depicted in below diagram**


```dot-svg
#p2p_local_chain_prunning.dot - local chain prunning
#
#notes: * to see image copy/paste to https://dreampuf.github.io/GraphvizOnline
# * image will be rendered by gatsby-remark-graphviz plugin in eosio docs.
digraph {
newrank=true #allows ranks inside subgraphs (important!)
compound=true #allows edges connecting nodes with subgraphs
graph [rankdir=LR]
node [style=filled, fillcolor=lightgray, shape=square, fixedsize=true, width=.55, fontsize=10]
edge [dir=both, arrowsize=.6, weight=100]
splines=false
subgraph cluster_chain {
label="Block Producers Peers"; labelloc="b"
graph [color=invis]
b0 [label="...", color=invis, style=""]
b1 [label="BP-A"]; b2 [label="BP-A\nPeer"]; b3 [label="BP-B\nPeer"]; b4 [label="BP-B"]
b5 [label="...", color=invis, style=""]
b0 -> b1 -> b2 -> b3 -> b4 -> b5
} //cluster_chain
} //digraph
```
+------+ +------+ +------+ +------+
-->| BP-A |---->| BP-A |------>| BP-B |---->| BP-B |
+------+ | Peer | | Peer | +------+
+------+ +------+
```

`BP-A` will send block at `p` and,

`BP-B` needs block at time `t` or otherwise will drop it.
`BP-A` will send block at `l` and, `BP-B` needs block at time `t` or otherwise will drop it.

If `BP-A`is producing 12 blocks as follows `b(lock) at t(ime) 1`, `bt 1.5`, `bt 2`, `bt 2.5`, `bt 3`, `bt 3.5`, `bt 4`, `bt 4.5`, `bt 5`, `bt 5.5`, `bt 6`, `bt 6.5` then `BP-B` needs `bt 6.5` by time `6.5` so it has `.5` to produce `bt 7`.

Please notice that the time of `bt 7` minus `.5` equals the time of `bt 6.5` therefore time `t` is the last block time of `BP-A` and when `BP-B` needs to start its first block.

## Example 1
`BP-A` has 50% e, m = 200ms, c = 200ms, n = 0ms, a = 250ms:
`BP-A` sends at (t-250ms) <-> `BP-A-Peer` processes for 200ms and sends at (t - 50ms) <-> `BP-B-Peer` processes for 200ms and sends at (t + 150ms) <-> arrive at `BP-B` 150ms too late.

## Example 2
`BP-A` has 40% e and m = 200ms, c = 200ms, n = 0ms, a = 300ms:
(t-300ms) <-> (+200ms) <-> (+200ms) <-> arrive at `BP-B` 100ms too late.

## Example 3
`BP-A` has 30% e and m = 200ms, c = 150ms, n = 0ms, a = 350ms:
(t-350ms) <-> (+150ms) <-> (+150ms) <-> arrive at `BP-B` with 50ms to spare.

## Example 4
`BP-A` has 25% e and m = 200ms, c = 125ms, n = 0ms, a = 375ms:
(t-375ms) <-> (+125ms) <-> (+125ms) <-> arrive at `BP-B` with 125ms to spare.

## Example 5
`BP-A` has 10% e and m = 200ms, c = 50ms, n = 0ms, a = 450ms:
(t-450ms) <-> (+50ms) <-> (+50ms) <-> arrive at `BP-B` with 350ms to spare.

## Example 6
`BP-A` has 10% e and m = 200ms, c = 50ms, n = 15ms, a = 450ms:
(t-450ms) <- +15ms -> (+50ms) <- +15ms -> (+50ms) <- +15ms -> `BP-B` <-> arrive with 305ms to spare.
A block is produced and sent when either it reaches `m` or `u` or `p`.

Starting in Leap 4.0, blocks are propagated after block header validation. This means instead of `BP-A Peer` & `BP-B Peer` taking `m` time to validate and forward a block it only takes a small number of milliseconds to verify the block header and then forward the block.

Starting in Leap 5.0, blocks in a round are started immediately after the completion of the previous block. Before 5.0, blocks were always started on `w` intervals and a node would "sleep" between blocks if needed. In 5.0, the "sleeps" are all moved to the end of the block production round.

## Example 1: block arrives 110ms early
* Assuming zero network latency between all nodes.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 120, n = 0ms, h = 5ms, a = 10ms
* `BP-A` sends b1 at `t1-10ms` => `BP-A-Peer` processes `h=5ms`, sends at `t-5ms` => `BP-B-Peer` processes `h=5ms`, sends at `t-0ms` => arrives at `BP-B` at `t`.
* `BP-A` starts b2 at `t1-10ms`, sends b2 at `t2-20ms` => `BP-A-Peer` processes `h=5ms`, sends at `t2-15ms` => `BP-B-Peer` processes `h=5ms`, sends at `t2-10ms` => arrives at `BP-B` at `t2-10ms`.
* `BP-A` starts b3 at `t2-20ms`, ...
* `BP-A` starts b12 at `t11-110ms`, sends b12 at `t12-120ms` => `BP-A-Peer` processes `h=5ms`, sends at `t12-115ms` => `BP-B-Peer` processes `h=5ms`, sends at `t12-110ms` => arrives at `BP-B` at `t12-110ms`

## Example 2: block arrives 80ms early
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 150ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 240, n = 0ms/150ms, h = 5ms, a = 20ms
* `BP-A` sends b1 at `t1-20ms` => `BP-A-Peer` processes `h=5ms`, sends at `t-15ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t+140ms` => arrives at `BP-B` at `t+140ms`.
* `BP-A` starts b2 at `t1-20ms`, sends b2 at `t2-40ms` => `BP-A-Peer` processes `h=5ms`, sends at `t2-35ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t2+120ms` => arrives at `BP-B` at `t2+120ms`.
* `BP-A` starts b3 at `t2-40ms`, ...
* `BP-A` starts b12 at `t11-220ms`, sends b12 at `t12-240ms` => `BP-A-Peer` processes `h=5ms`, sends at `t12-235ms` =(150ms)> `BP-B-Peer` processes `h=5ms`, sends at `t12-80ms` => arrives at `BP-B` at `t12-80ms`

## Example 3: block arrives 16ms late and is dropped
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 200ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assuming blocks do not reach `m` and therefore take `w - a` time to produce.
* Assume block completion including signing takes zero time.
* `BP-A` has e = 204, n = 0ms/200ms, h = 10ms, a = 17ms
* `BP-A` sends b1 at `t1-17ms` => `BP-A-Peer` processes `h=10ms`, sends at `t-7ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t+203ms` => arrives at `BP-B` at `t+203ms`.
* `BP-A` starts b2 at `t1-17ms`, sends b2 at `t2-34ms` => `BP-A-Peer` processes `h=10ms`, sends at `t2-24ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t2+186ms` => arrives at `BP-B` at `t2+186ms`.
* `BP-A` starts b3 at `t2-34ms`, ...
* `BP-A` starts b12 at `t11-187ms`, sends b12 at `t12-204ms` => `BP-A-Peer` processes `h=10ms`, sends at `t12-194ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t12+16ms` => arrives at `BP-B` at `t12+16ms`

## Example 4: full blocks are produced early
* Assuming zero network latency between `BP-A` & `BP-A Peer` and between `BP-B Peer` & `BP-B`.
* Assuming 200ms network latency between `BP-A Peer` & `BP-B Peer`.
* Assume all blocks are full as there are enough queued up unapplied transactions ready to fill all blocks.
* Assume a block can be produced with 200ms worth of transactions in 225ms worth of time. There is overhead for producing the block.
* `BP-A` has e = 120, m = 200ms, n = 0ms/200ms, h = 10ms, a = 10ms
* `BP-A` sends b1 at `t1-275s` => `BP-A-Peer` processes `h=10ms`, sends at `t-265ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t-55ms` => arrives at `BP-B` at `t-55ms`.
* `BP-A` starts b2 at `t1-275ms`, sends b2 at `t2-550ms (t1-50ms)` => `BP-A-Peer` processes `h=10ms`, sends at `t2-540ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t2-330ms` => arrives at `BP-B` at `t2-330ms`.
* `BP-A` starts b3 at `t2-550ms`, ...
* `BP-A` starts b12 at `t11-3025ms`, sends b12 at `t12-3300ms` => `BP-A-Peer` processes `h=10ms`, sends at `t12-3290ms` =(200ms)> `BP-B-Peer` processes `h=10ms`, sends at `t12-3080ms` => arrives at `BP-B` at `t12-3080ms`

## Example 7
Example world-wide network:`BP-A`has 10% e and m = 200ms, c = 50ms, n = 15ms/250ms, a = 450ms:
(t-450ms) <- +15ms -> (+50ms) <- +250ms -> (+50ms) <- +15ms -> `BP-B` <-> arrive with 70ms to spare.

Running wasm-runtime=eos-vm-jit eos-vm-oc-enable on relay node will reduce the validation time.
17 changes: 3 additions & 14 deletions docs/01_nodeos/03_plugins/producer_plugin/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,20 +72,9 @@ Config Options for eosio::producer_plugin:
can extend during low usage (only
enforced subjectively; use 1000 to not
enforce any limit)
--produce-time-offset-us arg (=0) Offset of non last block producing time
in microseconds. Valid range 0 ..
-block_time_interval.
--last-block-time-offset-us arg (=-200000)
Offset of last block producing time in
microseconds. Valid range 0 ..
-block_time_interval.
--cpu-effort-percent arg (=80) Percentage of cpu block production time
used to produce block. Whole number
percentages, e.g. 80 for 80%
--last-block-cpu-effort-percent arg (=80)
Percentage of cpu block production time
used to produce last block. Whole
number percentages, e.g. 80 for 80%
--produce-block-offset-ms arg (=450) The minimum time to reserve at the end
of a production round for blocks to
propagate to the next block producer.
--max-block-cpu-usage-threshold-us arg (=5000)
Threshold of CPU block production to
consider block full; when within
Expand Down
2 changes: 1 addition & 1 deletion libraries/chain/include/eosio/chain/config.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ const static uint32_t default_max_inline_action_size = 512 * 102
const static uint16_t default_max_inline_action_depth = 4;
const static uint16_t default_max_auth_depth = 6;
const static uint32_t default_sig_cpu_bill_pct = 50 * percent_1; // billable percentage of signature recovery
const static uint32_t default_block_cpu_effort_pct = 90 * percent_1; // percentage of block time used for producing block
const static uint32_t default_produce_block_offset_ms = 450;
const static uint16_t default_controller_thread_pool_size = 2;
const static uint32_t default_max_variable_signature_length = 16384u;
const static uint32_t default_max_action_return_value_size = 256;
Expand Down
18 changes: 9 additions & 9 deletions plugins/producer_plugin/include/eosio/producer_plugin/block_timing_util.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,28 +42,28 @@ namespace block_timing_util {
// In the past, a producer would always start a block `config::block_interval_us` ahead of its block time. However,
// it causes the last block in a block production round being released too late for the next producer to have
// received it and start producing on schedule. To mitigate the problem, we leave no time gap in block producing. For
// example, given block_interval=500 ms and cpu effort=400 ms, assuming the our round start at time point 0; in the
// example, given block_interval=500 ms and cpu effort=400 ms, assuming our round starts at time point 0; in the
// past, the block start time points would be at time point -500, 0, 500, 1000, 1500, 2000 .... With this new
// approach, the block time points would become -500, -100, 300, 700, 1100 ...
inline fc::time_point production_round_block_start_time(uint32_t cpu_effort_us, chain::block_timestamp_type block_time) {
inline fc::time_point production_round_block_start_time(fc::microseconds cpu_effort, chain::block_timestamp_type block_time) {
uint32_t block_slot = block_time.slot;
uint32_t production_round_start_block_slot =
(block_slot / chain::config::producer_repetitions) * chain::config::producer_repetitions;
uint32_t production_round_index = block_slot % chain::config::producer_repetitions;
return chain::block_timestamp_type(production_round_start_block_slot - 1).to_time_point() +
fc::microseconds(cpu_effort_us * production_round_index);
fc::microseconds(cpu_effort.count() * production_round_index);
}

inline fc::time_point calculate_producing_block_deadline(uint32_t cpu_effort_us, chain::block_timestamp_type block_time) {
auto estimated_deadline = production_round_block_start_time(cpu_effort_us, block_time) + fc::microseconds(cpu_effort_us);
inline fc::time_point calculate_producing_block_deadline(fc::microseconds cpu_effort, chain::block_timestamp_type block_time) {
auto estimated_deadline = production_round_block_start_time(cpu_effort, block_time) + cpu_effort;
auto now = fc::time_point::now();
if (estimated_deadline > now) {
return estimated_deadline;
} else {
// This could only happen when the producer stop producing and then comes back alive in the middle of its own
// production round. In this case, we just use the hard deadline.
const auto hard_deadline = block_time.to_time_point() - fc::microseconds(chain::config::block_interval_us - cpu_effort_us);
return std::min(hard_deadline, now + fc::microseconds(cpu_effort_us));
const auto hard_deadline = block_time.to_time_point() - fc::microseconds(chain::config::block_interval_us - cpu_effort.count());
return std::min(hard_deadline, now + cpu_effort);
}
}

Expand Down Expand Up @@ -118,7 +118,7 @@ namespace block_timing_util {
// Return the *next* block start time according to its block time slot.
// Returns empty optional if no producers are in the active_schedule.
// block_num is only used for watermark minimum offset.
inline std::optional<fc::time_point> calculate_producer_wake_up_time(uint32_t cpu_effort_us, uint32_t block_num,
inline std::optional<fc::time_point> calculate_producer_wake_up_time(fc::microseconds cpu_effort, uint32_t block_num,
const chain::block_timestamp_type& ref_block_time,
const std::set<chain::account_name>& producers,
const std::vector<chain::producer_authority>& active_schedule,
Expand All @@ -141,7 +141,7 @@ namespace block_timing_util {
return {};
}

return production_round_block_start_time(cpu_effort_us, chain::block_timestamp_type(wake_up_slot));
return production_round_block_start_time(cpu_effort, chain::block_timestamp_type(wake_up_slot));
}

} // namespace block_timing_util
Expand Down
5 changes: 3 additions & 2 deletions plugins/producer_plugin/include/eosio/producer_plugin/producer_plugin.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ class producer_plugin : public appbase::plugin<producer_plugin> {
struct runtime_options {
std::optional<int32_t> max_transaction_time;
std::optional<int32_t> max_irreversible_block_age;
std::optional<int32_t> cpu_effort_us;
// minimum time to reserve at the end of a production round for blocks to propagate to the next block producer.
std::optional<int32_t> produce_block_offset_ms;
std::optional<int32_t> subjective_cpu_leeway_us;
std::optional<uint32_t> greylist_limit;
};
Expand Down Expand Up @@ -196,7 +197,7 @@ class producer_plugin : public appbase::plugin<producer_plugin> {

} //eosio

FC_REFLECT(eosio::producer_plugin::runtime_options, (max_transaction_time)(max_irreversible_block_age)(cpu_effort_us)(subjective_cpu_leeway_us)(greylist_limit));
FC_REFLECT(eosio::producer_plugin::runtime_options, (max_transaction_time)(max_irreversible_block_age)(produce_block_offset_ms)(subjective_cpu_leeway_us)(greylist_limit));
FC_REFLECT(eosio::producer_plugin::greylist_params, (accounts));
FC_REFLECT(eosio::producer_plugin::whitelist_blacklist, (actor_whitelist)(actor_blacklist)(contract_whitelist)(contract_blacklist)(action_blacklist)(key_blacklist) )
FC_REFLECT(eosio::producer_plugin::integrity_hash_information, (head_block_id)(integrity_hash))
Expand Down
Loading

0 comments on commit d76783c

Please sign in to comment.