Support for Events or Pub/Sub

[tannerlinsley]

After discussing a few awesome ideas with @esjewett, I feel that there is a need to think about some integration points for crossfilter. One in particular is some way to know externally if the state or data inside a crossfilter has changed. Adding, removing, and filtering data are the three actions that come to mind.

In our current in-house solution, we are following up all of our calls to these crossfilter methods with a pup-sub like system. This works great for us, but breaks down when anything outside of our system (plugins, reductio, etc) call these methods for us.

I think it would be much easier to support some type of pub/sub support to use as an integration point.

The specific use case that sparked this conversation is my obsession with post-aggregations (think change stats for certain assessors eg. minChange) that can't be performed inside reduce functions, and must be applied after data is done changing (after any significant event in crossfilter).

Thoughts?

[tannerlinsley]

For reference: crossfilter/reductio#25

[derekperkins]

+:100: What would be the integration path? Would there be an extra function along the lines of enableCallbacks or is the performance hit when there are no listeners insignificant?

[tannerlinsley]

I don't think it would affect crossfilter performance almost at all. Obviously you would be able to do what ever you want in those callbacks, so you personally could make it slow, but this is simply "Hi, I'm crossfilter. Just so you know, I just added/removed/filtered records." Possibly one liners here and there throughout the code to trigger those callbacks.

[tannerlinsley]

There is also the question of how those callbacks would be registered. I'm sure we've all used tons of ways to register and deregister callbacks. I personally enjoy when an addCallback type function returns a function that in turn deregisters it. Much like angular.js $scope events.

[tannerlinsley]

At this point, @esjewett also mentioned just starting with a high level onChange event to handle everything, and that as soon as we start popping in integration points, things could get crazy going forward unless they're well structured and absolutely needed.

[derekperkins]

I'd say go ahead and submit a PR so we can use that as a base for discussion. If you and @esjewett are behind it, let's get started.

[esjewett]

PR would make sense. Want to get @gordonwoodhull's input as well. As far as the PR, I would recommend invoking the callback at the end of crossfilter.add, crossfilter.remove, and dimension.filter methods rather than trying to plug into the internal Crossfilter listener system.

[tannerlinsley]

@esjewett, I agree. The internal system is very fitted to crossfilter.
Might be hard to reason about for our purposes. I'll work up a PR by the
end of the day.

[gordonwoodhull]

This fits into the goal of making crossfilter more transparent. I appreciate that the original authors wanted to keep the API small, but it's kind of hard to use crossfilter as the point of integration between different modules which don't know about each other. And we naturally end up using it that way.

Obviously the callbacks should report the major operation, rather than row-by-row changes.

The internal listeners look like they do almost what we want, but they seem to do a bit more - in particular it looks like the filter listener is something quite different. I agree it's probably better to separate these.