Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix missing transaction in light mempool bug #1239

Merged

Conversation

Shourya742
Copy link
Contributor

Closes: #840

Copy link

codecov bot commented Oct 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 25.13%. Comparing base (66b1807) to head (7cc0424).
Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1239      +/-   ##
==========================================
+ Coverage   19.36%   25.13%   +5.76%     
==========================================
  Files         164       20     -144     
  Lines       10812     1134    -9678     
==========================================
- Hits         2094      285    -1809     
+ Misses       8718      849    -7869     
Flag Coverage Δ
binary_codec_sv2-coverage ?
binary_serde_sv2-coverage ?
binary_sv2-coverage ?
bip32_derivation-coverage ?
buffer_sv2-coverage ?
codec_sv2-coverage ?
common_messages_sv2-coverage ?
const_sv2-coverage ?
error_handling-coverage ?
framing_sv2-coverage ?
jd_client-coverage ?
jd_server-coverage ?
job_declaration_sv2-coverage ?
key-utils-coverage ?
mining-coverage ?
mining_device-coverage ?
mining_proxy_sv2-coverage ?
noise_sv2-coverage ?
pool_sv2-coverage ?
protocols ?
roles ?
roles_logic_sv2-coverage ?
sv1-mining-device-coverage ?
sv2_ffi-coverage ?
template_distribution_sv2-coverage ?
translator_sv2-coverage ?
utils 25.13% <ø> (ø)
v1-coverage ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Shourya742 Shourya742 changed the title [WIP] Fix missing transaction in light mempool bug Fix missing transaction in light mempool bug Nov 2, 2024
@Shourya742 Shourya742 marked this pull request as ready for review November 2, 2024 15:57
@GitGab19
Copy link
Collaborator

GitGab19 commented Nov 3, 2024

@Shourya742 I had a quick look at this solution and I have two major points to discuss:

  • You remove all the txs contained in an old job from the mempool --> probably the majority of the txs contained in the new job will be the same --> it doesn't make too much sense to remove and add them again
  • To properly remove old txs, you transform the mempool from txid to short_id (which doesn't scale well as approach and it's already a bottleneck we encounter when we receive a DeclareMiningJob message.

Have you explored the idea about not removing the old txs, but simply updating the mempool (as it's already done) and adding the txs contained in the last declared jobs (if they are not already there in the mempool after the update)?
Does it make sense to you? If not, which could be the issues?

@Shourya742
Copy link
Contributor Author

Shourya742 commented Nov 4, 2024

@Shourya742 I had a quick look at this solution and I have two major points to discuss:

  • You remove all the txs contained in an old job from the mempool --> probably the majority of the txs contained in the new job will be the same --> it doesn't make too much sense to remove and add them again
  • To properly remove old txs, you transform the mempool from txid to short_id (which doesn't scale well as approach and it's already a bottleneck we encounter when we receive a DeclareMiningJob message.

Have you explored the idea about not removing the old txs, but simply updating the mempool (as it's already done) and adding the txs contained in the last declared jobs (if they are not already there in the mempool after the update)? Does it make sense to you? If not, which could be the issues?

Point 1: We are checking if the txid is not present in the new job. Based on this, we decrement the counter value rather than deleting everything from the old job.

Point 2: I didn’t know that—let me check on it.

I initially thought keeping the JDS mempool as lean as possible was a requirement. However, it does make sense to let transactions persist in the mempool without a specific logic to remove them once a new DMJ is anticipated. In my testing, while we have logic to remove transactions from the old job, it didn’t result in a significant reduction in the number of transactions deleted from the mempool. But if we’re okay with not keeping the JDS mempool lean, then we can follow the approach you suggested.

@Shourya742 Shourya742 force-pushed the 2024-10-missing-transaction-mempool branch from f244cf7 to 5d0462d Compare November 12, 2024 06:32
Copy link
Contributor

github-actions bot commented Nov 12, 2024

🐰 Bencher Report

Branch2024-10-missing-transaction-mempool
Testbedsv2
🚨 1 ALERT: Threshold Boundary Limit exceeded!
BenchmarkMeasure
Units
ViewBenchmark Result
(Result Δ%)
Lower Boundary
(Limit %)
Upper Boundary
(Limit %)
client_sv2_open_channel_serialize_deserializeL2 Accesses
accesses
📈 plot
🚨 alert
🚷 threshold
87.00
(+17.28%)
85.01
(102.34%)
Click to view all benchmark results
BenchmarkEstimated CyclesBenchmark Result
estimated cycles
(Result Δ%)
Upper Boundary
estimated cycles
(Limit %)
InstructionsBenchmark Result
instructions
(Result Δ%)
Upper Boundary
instructions
(Limit %)
L1 AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
L2 AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
RAM AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
client_sv2_handle_message_common📈 view plot
🚷 view threshold
2,053.00
(-1.86%)
2,204.66
(93.12%)
📈 view plot
🚷 view threshold
473.00
(-0.09%)
490.77
(96.38%)
📈 view plot
🚷 view threshold
738.00
(+0.27%)
760.99
(96.98%)
📈 view plot
🚷 view threshold
4.00
(-27.07%)
14.28
(28.02%)
📈 view plot
🚷 view threshold
37.00
(-2.51%)
41.08
(90.07%)
client_sv2_handle_message_mining📈 view plot
🚷 view threshold
8,212.00
(+0.06%)
8,292.10
(99.03%)
📈 view plot
🚷 view threshold
2,137.00
(-0.03%)
2,143.08
(99.72%)
📈 view plot
🚷 view threshold
3,157.00
(-0.12%)
3,170.59
(99.57%)
📈 view plot
🚷 view threshold
38.00
(+6.55%)
42.49
(89.44%)
📈 view plot
🚷 view threshold
139.00
(-0.06%)
140.99
(98.59%)
client_sv2_mining_message_submit_standard📈 view plot
🚷 view threshold
6,213.00
(-1.04%)
6,389.35
(97.24%)
📈 view plot
🚷 view threshold
1,750.00
(-0.06%)
1,769.01
(98.93%)
📈 view plot
🚷 view threshold
2,553.00
(-0.04%)
2,579.84
(98.96%)
📈 view plot
🚷 view threshold
18.00
(+6.13%)
24.79
(72.60%)
📈 view plot
🚷 view threshold
102.00
(-1.90%)
107.02
(95.31%)
client_sv2_mining_message_submit_standard_serialize📈 view plot
🚷 view threshold
14,554.00
(-1.21%)
14,982.55
(97.14%)
📈 view plot
🚷 view threshold
4,694.00
(-0.02%)
4,713.01
(99.60%)
📈 view plot
🚷 view threshold
6,759.00
(+0.00%)
6,785.37
(99.61%)
📈 view plot
🚷 view threshold
47.00
(+8.05%)
53.45
(87.94%)
📈 view plot
🚷 view threshold
216.00
(-2.53%)
228.21
(94.65%)
client_sv2_mining_message_submit_standard_serialize_deserialize📈 view plot
🚷 view threshold
27,352.00
(-0.62%)
27,715.42
(98.69%)
📈 view plot
🚷 view threshold
10,585.00
(+0.07%)
10,634.07
(99.54%)
📈 view plot
🚷 view threshold
15,402.00
(+0.07%)
15,477.48
(99.51%)
📈 view plot
🚷 view threshold
87.00
(+7.14%)
89.46
(97.25%)
📈 view plot
🚷 view threshold
329.00
(-1.80%)
341.70
(96.28%)
client_sv2_open_channel📈 view plot
🚷 view threshold
4,303.00
(-2.20%)
4,628.48
(92.97%)
📈 view plot
🚷 view threshold
1,461.00
(-0.03%)
1,478.77
(98.80%)
📈 view plot
🚷 view threshold
2,163.00
(+0.15%)
2,187.59
(98.88%)
📈 view plot
🚷 view threshold
8.00
(-11.27%)
17.84
(44.83%)
📈 view plot
🚷 view threshold
60.00
(-4.33%)
68.79
(87.22%)
client_sv2_open_channel_serialize📈 view plot
🚷 view threshold
13,858.00
(-1.49%)
14,482.08
(95.69%)
📈 view plot
🚷 view threshold
5,064.00
(-0.01%)
5,081.77
(99.65%)
📈 view plot
🚷 view threshold
7,328.00
(+0.03%)
7,355.16
(99.63%)
📈 view plot
🚷 view threshold
39.00
(+11.13%)
43.30
(90.08%)
📈 view plot
🚷 view threshold
181.00
(-3.51%)
199.53
(90.71%)
client_sv2_open_channel_serialize_deserialize📈 view plot
🚷 view threshold
22,569.00
(-0.41%)
22,902.65
(98.54%)
📈 view plot
🚷 view threshold
8,027.00
(+0.10%)
8,074.42
(99.41%)
📈 view plot
🚷 view threshold
11,669.00
(+0.04%)
11,755.38
(99.27%)
📈 view plot
🚨 view alert
🚷 view threshold
87.00
(+17.28%)
85.01
(102.34%)
📈 view plot
🚷 view threshold
299.00
(-1.52%)
312.36
(95.72%)
client_sv2_setup_connection📈 view plot
🚷 view threshold
4,643.00
(-0.89%)
4,778.18
(97.17%)
📈 view plot
🚷 view threshold
1,502.00
(-0.03%)
1,519.77
(98.83%)
📈 view plot
🚷 view threshold
2,278.00
(-0.04%)
2,302.95
(98.92%)
📈 view plot
🚷 view threshold
11.00
(+19.73%)
15.61
(70.49%)
📈 view plot
🚷 view threshold
66.00
(-2.11%)
70.00
(94.29%)
client_sv2_setup_connection_serialize📈 view plot
🚷 view threshold
16,024.00
(-0.94%)
16,485.43
(97.20%)
📈 view plot
🚷 view threshold
5,963.00
(-0.01%)
5,980.77
(99.70%)
📈 view plot
🚷 view threshold
8,664.00
(+0.01%)
8,694.64
(99.65%)
📈 view plot
🚷 view threshold
44.00
(+9.85%)
53.73
(81.90%)
📈 view plot
🚷 view threshold
204.00
(-2.36%)
217.01
(94.01%)
client_sv2_setup_connection_serialize_deserialize📈 view plot
🚷 view threshold
35,376.00
(-0.49%)
35,742.20
(98.98%)
📈 view plot
🚷 view threshold
14,855.00
(+0.06%)
14,903.44
(99.67%)
📈 view plot
🚷 view threshold
21,821.00
(+0.05%)
21,912.93
(99.58%)
📈 view plot
🚷 view threshold
100.00
(+7.47%)
114.81
(87.10%)
📈 view plot
🚷 view threshold
373.00
(-1.65%)
385.02
(96.88%)
🐰 View full continuous benchmarking report in Bencher

Copy link
Contributor

github-actions bot commented Nov 12, 2024

🐰 Bencher Report

Branch2024-10-missing-transaction-mempool
Testbedsv1
Click to view all benchmark results
BenchmarkEstimated CyclesBenchmark Result
estimated cycles
(Result Δ%)
Upper Boundary
estimated cycles
(Limit %)
InstructionsBenchmark Result
instructions
(Result Δ%)
Upper Boundary
instructions
(Limit %)
L1 AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
L2 AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
RAM AccessesBenchmark Result
accesses
(Result Δ%)
Upper Boundary
accesses
(Limit %)
get_authorize📈 view plot
🚷 view threshold
8,447.00
(-0.12%)
8,829.57
(95.67%)
📈 view plot
🚷 view threshold
3,689.00
(-1.59%)
3,890.44
(94.82%)
📈 view plot
🚷 view threshold
5,157.00
(-1.90%)
5,457.35
(94.50%)
📈 view plot
🚷 view threshold
7.00
(-1.10%)
12.25
(57.16%)
📈 view plot
🚷 view threshold
93.00
(+2.85%)
96.24
(96.63%)
get_submit📈 view plot
🚷 view threshold
95,509.00
(+0.07%)
96,155.99
(99.33%)
📈 view plot
🚷 view threshold
59,346.00
(-0.21%)
59,814.82
(99.22%)
📈 view plot
🚷 view threshold
85,219.00
(-0.24%)
85,912.42
(99.19%)
📈 view plot
🚷 view threshold
42.00
(-7.72%)
65.89
(63.74%)
📈 view plot
🚷 view threshold
288.00
(+2.90%)
289.55
(99.47%)
get_subscribe📈 view plot
🚷 view threshold
7,928.00
(-1.10%)
8,383.20
(94.57%)
📈 view plot
🚷 view threshold
2,765.00
(-2.46%)
2,970.09
(93.09%)
📈 view plot
🚷 view threshold
3,848.00
(-2.90%)
4,150.11
(92.72%)
📈 view plot
🚷 view threshold
11.00
(-8.45%)
20.85
(52.75%)
📈 view plot
🚷 view threshold
115.00
(+0.79%)
120.16
(95.71%)
serialize_authorize📈 view plot
🚷 view threshold
12,270.00
(-0.03%)
12,650.51
(96.99%)
📈 view plot
🚷 view threshold
5,266.00
(-1.01%)
5,460.66
(96.44%)
📈 view plot
🚷 view threshold
7,325.00
(-1.27%)
7,619.38
(96.14%)
📈 view plot
🚷 view threshold
9.00
(+0.44%)
15.31
(58.77%)
📈 view plot
🚷 view threshold
140.00
(+1.87%)
143.69
(97.43%)
serialize_deserialize_authorize📈 view plot
🚷 view threshold
24,400.00
(-1.14%)
25,286.60
(96.49%)
📈 view plot
🚷 view threshold
9,838.00
(-0.54%)
10,054.46
(97.85%)
📈 view plot
🚷 view threshold
13,890.00
(-0.54%)
14,208.08
(97.76%)
📈 view plot
🚷 view threshold
30.00
(-13.65%)
44.21
(67.86%)
📈 view plot
🚷 view threshold
296.00
(-1.73%)
315.99
(93.67%)
serialize_deserialize_handle_authorize📈 view plot
🚷 view threshold
30,144.00
(-0.56%)
30,798.34
(97.88%)
📈 view plot
🚷 view threshold
12,015.00
(-0.56%)
12,224.90
(98.28%)
📈 view plot
🚷 view threshold
16,999.00
(-0.65%)
17,316.21
(98.17%)
📈 view plot
🚷 view threshold
60.00
(+7.73%)
68.16
(88.03%)
📈 view plot
🚷 view threshold
367.00
(-0.63%)
382.16
(96.03%)
serialize_deserialize_handle_submit📈 view plot
🚷 view threshold
126,463.00
(-0.01%)
127,200.55
(99.42%)
📈 view plot
🚷 view threshold
73,200.00
(-0.12%)
73,698.97
(99.32%)
📈 view plot
🚷 view threshold
104,913.00
(-0.15%)
105,667.64
(99.29%)
📈 view plot
🚷 view threshold
103.00
(-4.24%)
135.19
(76.19%)
📈 view plot
🚷 view threshold
601.00
(+0.82%)
605.46
(99.26%)
serialize_deserialize_handle_subscribe📈 view plot
🚷 view threshold
27,532.00
(-1.04%)
28,654.58
(96.08%)
📈 view plot
🚷 view threshold
9,584.00
(-0.66%)
9,789.53
(97.90%)
📈 view plot
🚷 view threshold
13,542.00
(-0.77%)
13,848.78
(97.78%)
📈 view plot
🚷 view threshold
68.00
(+7.10%)
75.99
(89.48%)
📈 view plot
🚷 view threshold
390.00
(-1.50%)
417.44
(93.43%)
serialize_deserialize_submit📈 view plot
🚷 view threshold
115,155.00
(-0.08%)
115,992.24
(99.28%)
📈 view plot
🚷 view threshold
68,060.00
(-0.07%)
68,513.49
(99.34%)
📈 view plot
🚷 view threshold
97,655.00
(-0.09%)
98,355.01
(99.29%)
📈 view plot
🚷 view threshold
63.00
(+0.04%)
79.05
(79.70%)
📈 view plot
🚷 view threshold
491.00
(+0.01%)
499.68
(98.26%)
serialize_deserialize_subscribe📈 view plot
🚷 view threshold
22,906.00
(-1.44%)
24,082.48
(95.11%)
📈 view plot
🚷 view threshold
8,143.00
(-0.73%)
8,346.21
(97.57%)
📈 view plot
🚷 view threshold
11,466.00
(-0.77%)
11,756.53
(97.53%)
📈 view plot
🚷 view threshold
34.00
(-8.95%)
44.69
(76.09%)
📈 view plot
🚷 view threshold
322.00
(-1.99%)
349.39
(92.16%)
serialize_submit📈 view plot
🚷 view threshold
99,910.00
(+0.05%)
100,525.28
(99.39%)
📈 view plot
🚷 view threshold
61,408.00
(-0.18%)
61,861.06
(99.27%)
📈 view plot
🚷 view threshold
88,075.00
(-0.22%)
88,761.37
(99.23%)
📈 view plot
🚷 view threshold
50.00
(+5.89%)
66.07
(75.68%)
📈 view plot
🚷 view threshold
331.00
(+1.99%)
334.03
(99.09%)
serialize_subscribe📈 view plot
🚷 view threshold
11,373.00
(-0.21%)
11,761.53
(96.70%)
📈 view plot
🚷 view threshold
4,118.00
(-1.53%)
4,316.21
(95.41%)
📈 view plot
🚷 view threshold
5,708.00
(-1.92%)
6,006.21
(95.03%)
📈 view plot
🚷 view threshold
13.00
(+1.16%)
20.66
(62.92%)
📈 view plot
🚷 view threshold
160.00
(+1.58%)
164.14
(97.48%)
🐰 View full continuous benchmarking report in Bencher

Copy link
Contributor

github-actions bot commented Nov 12, 2024

🐰 Bencher Report

Branch2024-10-missing-transaction-mempool
Testbedsv1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Upper Boundary
nanoseconds (ns)
(Limit %)
client-submit-serialize📈 view plot
🚷 view threshold
6,761.30
(-0.22%)
8,223.21
(82.22%)
client-submit-serialize-deserialize📈 view plot
🚷 view threshold
7,630.80
(-0.62%)
9,283.45
(82.20%)
client-submit-serialize-deserialize-handle/client-submit-serialize-deserialize-handle📈 view plot
🚷 view threshold
8,191.40
(-0.61%)
9,784.34
(83.72%)
client-sv1-authorize-serialize-deserialize-handle/client-sv1-authorize-serialize-deserialize-handle📈 view plot
🚷 view threshold
884.09
(-0.29%)
1,037.03
(85.25%)
client-sv1-authorize-serialize-deserialize/client-sv1-authorize-serialize-deserialize📈 view plot
🚷 view threshold
677.15
(-0.68%)
860.15
(78.72%)
client-sv1-authorize-serialize/client-sv1-authorize-serialize📈 view plot
🚷 view threshold
248.58
(+0.31%)
268.54
(92.57%)
client-sv1-get-authorize/client-sv1-get-authorize📈 view plot
🚷 view threshold
158.03
(+0.32%)
166.44
(94.95%)
client-sv1-get-submit📈 view plot
🚷 view threshold
6,548.20
(-0.03%)
8,175.70
(80.09%)
client-sv1-get-subscribe/client-sv1-get-subscribe📈 view plot
🚷 view threshold
277.86
(-0.45%)
322.25
(86.23%)
client-sv1-subscribe-serialize-deserialize-handle/client-sv1-subscribe-serialize-deserialize-handle📈 view plot
🚷 view threshold
730.99
(-2.36%)
913.01
(80.06%)
client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize📈 view plot
🚷 view threshold
597.50
(-1.66%)
729.21
(81.94%)
client-sv1-subscribe-serialize/client-sv1-subscribe-serialize📈 view plot
🚷 view threshold
208.34
(+0.65%)
231.26
(90.09%)
🐰 View full continuous benchmarking report in Bencher

Copy link
Contributor

github-actions bot commented Nov 12, 2024

🐰 Bencher Report

Branch2024-10-missing-transaction-mempool
Testbedsv2
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Upper Boundary
nanoseconds (ns)
(Limit %)
client_sv2_handle_message_common📈 view plot
🚷 view threshold
43.74
(-2.27%)
48.67
(89.88%)
client_sv2_handle_message_mining📈 view plot
🚷 view threshold
75.13
(-0.74%)
88.37
(85.02%)
client_sv2_mining_message_submit_standard📈 view plot
🚷 view threshold
14.65
(+0.00%)
14.78
(99.16%)
client_sv2_mining_message_submit_standard_serialize📈 view plot
🚷 view threshold
273.43
(+0.26%)
308.17
(88.73%)
client_sv2_mining_message_submit_standard_serialize_deserialize📈 view plot
🚷 view threshold
596.31
(+1.18%)
703.64
(84.75%)
client_sv2_open_channel📈 view plot
🚷 view threshold
148.60
(+0.99%)
154.17
(96.39%)
client_sv2_open_channel_serialize📈 view plot
🚷 view threshold
278.07
(-2.38%)
324.12
(85.79%)
client_sv2_open_channel_serialize_deserialize📈 view plot
🚷 view threshold
371.09
(-1.18%)
431.30
(86.04%)
client_sv2_setup_connection📈 view plot
🚷 view threshold
162.06
(-0.12%)
188.72
(85.87%)
client_sv2_setup_connection_serialize📈 view plot
🚷 view threshold
464.73
(-0.39%)
601.38
(77.28%)
client_sv2_setup_connection_serialize_deserialize📈 view plot
🚷 view threshold
1,028.80
(+5.63%)
1,208.40
(85.14%)
🐰 View full continuous benchmarking report in Bencher

@stratum-mining stratum-mining deleted a comment from github-actions bot Nov 12, 2024
@stratum-mining stratum-mining deleted a comment from github-actions bot Nov 12, 2024
@stratum-mining stratum-mining deleted a comment from github-actions bot Nov 12, 2024
@stratum-mining stratum-mining deleted a comment from github-actions bot Nov 12, 2024
Copy link
Collaborator

@GitGab19 GitGab19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I would change the [mempool_update_interval] parameter to 0.1 secs as we tested together.

Copy link
Collaborator

@GitGab19 GitGab19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I just left a minor comment. I would say that we could add an integration test for this scenario in the future. I remember we were tracking these issues to be tested by the new integration tests framework somewhere, but I can't find it anymore (cc @plebhash)

roles/jd-server/src/lib/job_declarator/mod.rs Show resolved Hide resolved
@Shourya742 Shourya742 force-pushed the 2024-10-missing-transaction-mempool branch from 05cb2a8 to b907e7d Compare November 13, 2024 10:33
@Shourya742 Shourya742 force-pushed the 2024-10-missing-transaction-mempool branch from b907e7d to 7cc0424 Compare November 13, 2024 15:37
@rrybarczyk rrybarczyk merged commit dee3cae into stratum-mining:main Nov 13, 2024
38 checks passed
@plebhash plebhash mentioned this pull request Nov 13, 2024
@plebhash
Copy link
Collaborator

LGTM. I just left a minor comment. I would say that we could add an integration test for this scenario in the future. I remember we were tracking these issues to be tested by the new integration tests framework somewhere, but I can't find it anymore (cc @plebhash)

it's #1077

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom transactions from JDC removed from mempool during update
4 participants