Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Use NetMQ poller for all receive sockets #11

Merged
merged 3 commits into from
Jan 5, 2023

Conversation

glopesdev
Copy link
Member

This PR ensures that the event infrastructure of the NetMQPoller class is used for all receive sockets. Rather than using blocking calls to receive methods, a subscription is made to the ReceiveReady event. This also ensures that cancellation can always break out of the listening loop.

Fixes #10

@glopesdev glopesdev added the fix Pull request that fixes an issue label Jan 3, 2023
@glopesdev glopesdev added this to the 0.1.0 milestone Jan 3, 2023
@glopesdev glopesdev requested a review from RoboDoig January 3, 2023 19:46
@RoboDoig
Copy link
Collaborator

RoboDoig commented Jan 4, 2023

According to the docs at https://netmq.readthedocs.io/en/latest/poller/ - use of a poller is slower than directly receiving from a socket. If it is significantly slower we might want to consider (in future) using the suggested fix of limiting the number of messages that can be fetched in one batch on a socket.

@glopesdev
Copy link
Member Author

glopesdev commented Jan 4, 2023

Yes, I noticed the underlying code is more complex. However, it seems there is really no way to reliably be able to cancel a blocking call to a socket asynchronously without using poller (I tried a lot! It boils down to specific implementation details of NetMQ...).

Bonsai workflows always need to retain their ability to be asynchronously cancelled, otherwise the whole program would hang and you wouldn't be able to compose sources/sinks with operators such as TakeUntil/SubscribeWhen, etc.

I also noticed that for some reason I was getting a few dropped messages at the beginning when doing simple block receive on pull sockets without using a poller.

Once I switched to poller, both cancellation and message drop issues were resolved and everything started working fine.

We can always optimize in later versions, so probably would be best to stabilise the API for the first release. It's hard to benchmark performance outside of specific applications anyway, so I would wait until we use this version in a specific project to see where we are in this respect.

However, I do like the suggestion in the manual to do a batch while-loop with the TryReceive* methods instead of just handling one request per event, so will try to do that for now.

@glopesdev glopesdev merged commit eeb2de8 into bonsai-rx:main Jan 5, 2023
@glopesdev glopesdev deleted the api-cleanup branch January 5, 2023 10:10
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
fix Pull request that fixes an issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Push-pull pattern can drop messages while initializing
2 participants