Updates for parallel processing #218
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
peterbroadhurst
requested review from
vdamle,
jimthematrix and
awrichar
as code owners
Codecov Report
@@ Coverage Diff @@
## main #218 +/- ##
==========================================
+ Coverage 97.15% 97.20% +0.05%
==========================================
Files 70 70
Lines 7579 7661 +82
==========================================
+ Hits 7363 7447 +84
+ Misses 165 164 -1
+ Partials 51 50 -1
Continue to review full report at Codecov.
|
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
peterbroadhurst
force-pushed
the
parallel-addrbook
branch
from
aed3236 to
dcd0b7a
Compare
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
peterbroadhurst
force-pushed
the
parallel-addrbook
branch
from
dcd0b7a to
284c37c
Compare
# This is the 1st commit message: Add concurrency debug Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io> # This is the commit message hyperledger#2: Debug time in OnMessage Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
peterbroadhurst
force-pushed
the
parallel-addrbook
branch
from
3eecec2 to
c4599ab
Compare
…g retry Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
peterbroadhurst
changed the title
vdamle
reviewed
May 17, 2022
vdamle
reviewed
May 17, 2022
| @@ -133,6 +150,25 @@ func (p *txnProcessor) Init(rpc eth.RPCClient) { | |||
| if p.conf.HDWalletConf.URLTemplate != "" { | |||
| p.hdwallet = newHDWallet(&p.conf.HDWalletConf) | |||
| } | |||
| p.concurrencySlots = make(chan bool, p.conf.SendConcurrency) | |||
There was a problem hiding this comment.
After a discussion with @peterbroadhurst , I note that this needs to be setup in Init() because of the way this function and the caller of this function NewKafkaBridge() is called:
firefly-ethconnect/cmd/ethconnect.go
Lines 172 to 173 in 442cfb3
The config is set after the constructor is invoked, so setting it in the constructor doesn't have the intended effect.
vdamle
approved these changes
May 17, 2022
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Three related changes from analysis of some performance tests:
The
sendConcurrencysetting is not currently providing the full concurrency support it was intended to. This is because theconcurrencySlotschannel was being allocated in the constructor of theTxnProcessorwhich is beforeSetConfis called. So we need to allocated it inInit(). This results in us using pipelining between theSendand in-flight nonce allocation, but not performing multipleSendcalls in parallel.In the case that a
Sendfails due to an intermittent error, such as a timeout, this is problem for end-users because they do not know if the transaction was submitted or not. However, in cases where EthConnect is in control of thenonceit is safe for EthConnect to retry. So this PR introduces an option for retries, and enables these by default (with a small number - 5) to deal with glitches without pushing those errors all the way back to the requester.Currently the addressbook lookup is performed on the single-threaded processing, before we dispatch a concurrent worker to the
inflightaction. As there is a JSON/RPC healthcheck call that is part of this (net_version) this is inefficient. So this PR moves it to the concurrent worker, afterinflightis generated.