Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Inflight reset #206

Merged
merged 2 commits into from
Mar 18, 2022
Merged

Inflight reset #206

merged 2 commits into from
Mar 18, 2022

Conversation

peterbroadhurst
Copy link
Contributor

Currently every time FireFly starts, it resets the EthConnect stream with a PATCH in order to apply any changes.
I observed a problem where a stream did not continue sending events after such an event.

I was able to produce a problem running two simple curl loops, where one stream stalled:

1. Sending messages with confirm

while [ 1 ]; do echo POSTING $(date); curl -s -X POST -d '{"data":[{"value":"test"}]}' -H "Content-type: application/json" 'localhost:5000/api/v1/namespaces/default/messages/broadcast?confirm' | jq; sleep 1; done

2. Resetting the stream

while [ 1 ]; do echo RESET $(date); curl -s -X PATCH -d @/tmp/ev.json  -H "Content-type: application/json" 0.0.0.0:5102/eventstreams/es-9f5df9e3-563e-4f11-7750-5d147ccf0231; sleep 1; done

There was not anything in the logs that really showed why the stream had stopped distributing data, and nothing in a goroutine dump. So I spent some time on code inspection, and made the changes in this PR.

  • Reset the InFlight count to zero on update
    • This seemed to be a straight bug, and after fixing this I was not able to reproduce the stall
  • Add better logging in case it happens again
    • Particularly per-polling loop debug when InFlight > 0
  • Do not restart a stream if a PATCH does not result in any change

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@codecov-commenter
Copy link

Codecov Report

Merging #206 (6857002) into main (6f5a277) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #206   +/-   ##
=======================================
  Coverage   97.20%   97.21%           
=======================================
  Files          58       58           
  Lines        6902     6926   +24     
=======================================
+ Hits         6709     6733   +24     
  Misses        147      147           
  Partials       46       46           
Impacted Files Coverage Δ
ethconnect/internal/events/eventstream.go 96.81% <0.00%> (+0.21%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f5a277...6857002. Read the comment docs.

@peterbroadhurst peterbroadhurst merged commit 2e95ecd into hyperledger:main Mar 18, 2022
@peterbroadhurst peterbroadhurst deleted the inflight-reset branch March 18, 2022 16:26
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants