Comparison with dplyr and Stata

[matthieugomez]

This adds a very simple table to compare the main functions in DataFrames with dplyr and Stata.
I focused on the main functions to make it as simple as possible.

[bkamins]

Thank you for working on it. I have left some comments.

[pdeffebach]

Thanks for this! It's super helpful.

I think we could make this a bit more biased towards DataFrames. For instance, working with variables rather than literals in a comparison between DataFrames and dplyr. Similarly, we should show off describe more.

[matthieugomez]

I've answered the comments and added two examples of functions that return dataframes rather than vectors.

[bkamins]

Thank you for the fixes. Let us wait for @nalimilan to have a look at the PR.

[pdeffebach]

Thanks!

In general the html output is a bit awkward. Lots of lines are cut off. Bit I'm not sure what we can do about that.

[bkamins]

Maybe we should switch from a table to a list of lists? Not sure what would look better though

[matthieugomez]

We could split the comparison into a dplyr and stata one, which would make each table less wide. Moreover, in reality people tend to be interested in either comparison, not both at the same time.

[bkamins]

Good point - agreed. It will create some duplication but I do not think it is a problem (we can make separate sections for them).

[matthieugomez]

Done. This led me to add a Stata specific operation: transform certain rows. Makes me think that it would be nice to define some kind of replace signature to make it easier.

[bkamins]

Makes me think that it would be nice to define some kind of replace signature to make it easier.

Indeed, especially as ifelse.(y .>= 1, 1, x)) will fail with missing values (our usual scenario). Can you open an Issue please with what you think would be good (or just a call to add it so that we keep track of it - I have a remote memory that we already discussed it, but it might have been lost so it is better to have an issue for it, as it is a common operation)

[pdeffebach]

I think passmissing(ifelse).(...) is good enough to include here.

[bkamins]

Unfortunately it is incorrect. You have to write ifelse.(.!ismissing.(y) .& y .>= 1, 1, x).

Using passmissing(ifelse) will produce missing in the result instead of retaining the value stored in x.

[bkamins]

This is the reason that I am on board with efforts to make handling missing easier (but I am too busy with other PRs now to sit down to it seriously). What were the latest ideas to solve this issue (it was a macro if I recall correctly - right?).

or in the original code:

ifelse.(coalesce.(y .>= 1, false), 1, x)

which feels more idiomatic.

[matthieugomez]

I think it’s #2211

[bkamins]

@nalimilan - ok to merge?

[bkamins]

@nalimilan - I would merge it if you do not have additional comments.

[bkamins]

So - is this good to merge?

[nalimilan]

Just waiting for Travis... :-p

[nalimilan]

Thanks!

[nalimilan]

The new page doesn't show up in the manual. I think you need to add it to the list too. Also there are a few issues with some tables, see https://juliadata.github.io/DataFrames.jl/latest/man/comparisons/

[nalimilan]

Maybe it would be useful to add an example of a simple row-wise function using gen in Stata? For example it's not obvious that ByRow is also useful for things like :x => ByRow(x -> x^2) => :x².