Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Implemented DataFrame.lookup #1785

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Implemented DataFrame.lookup #1785

wants to merge 12 commits into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Sep 22, 2020

This PR proposes DataFrame.lookup.

>>> kdf = ks.DataFrame({'A': [3, 4, 5, 6, 7],
...                     'B': [10.0, 20.0, 30.0, 40.0, 50.0],
...                     'C': ['a', 'b', 'c', 'd', 'e']})
>>> kdf
   A     B  C
0  3  10.0  a
1  4  20.0  b
2  5  30.0  c
3  6  40.0  d
4  7  50.0  e

>>> kdf.lookup([0], ["C"])
array(['a'], dtype=object)

>>> kdf.lookup([2, 3], ["A", "B"])
array([ 5., 40.])

itholic
itholic commented Sep 22, 2020
View reviewed changes
@itholic
Copy link
Contributor Author

itholic commented Sep 24, 2020

Thanks, @HyukjinKwon

ueshin
ueshin reviewed Sep 25, 2020
View reviewed changes
@itholic
Copy link
Contributor Author

itholic commented Sep 26, 2020

Thanks, @ueshin

HyukjinKwon
HyukjinKwon reviewed Sep 29, 2020
View reviewed changes
databricks/koalas/frame.py
raise ValueError("Row labels must have same size as column labels")
lookups = [
self.loc[row_label, col_label]
for row_label, col_label in zip(row_labels, col_labels)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean we launch jobs as many as labels?

xinrong-meng
xinrong-meng reviewed Nov 24, 2020
View reviewed changes
databricks/koalas/frame.py Outdated
@@ -10277,7 +10277,7 @@ def lookup(self, row_labels, col_labels) -> np.ndarray:
lookups = [
self.loc[row_label, col_label] for row_label, col_label in zip(row_labels, col_labels)
]
return Series(lookups).to_numpy()
return np.asarray(pd.Series(lookups))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why did we change it to np.asarray(pd.Series(lookups))?

@xinrong-meng
Copy link
Contributor

https://issues.apache.org/jira/browse/SPARK-36434

# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@ueshin ueshin ueshin left review comments

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@xinrong-meng xinrong-meng xinrong-meng left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@itholic @xinrong-meng @ueshin @HyukjinKwon