Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

What do you think about Selector(response).xpath().map() ? #92

Open
pawelmhm opened this issue Jun 22, 2017 · 3 comments
Open

What do you think about Selector(response).xpath().map() ? #92

pawelmhm opened this issue Jun 22, 2017 · 3 comments

Comments

@pawelmhm
Copy link

pawelmhm commented Jun 22, 2017

Sometimes I'd like to apply some function after extracting something, and I do something like this:

In[32]: map(json.loads, sel.xpath("//@data-p13n-asin-metadata").extract())

Out[32]: [{u'asin': u'B00Y2863GQ', u'price': 221.99, u'ref': u'pd_bxgy_75_1_1'},
 {u'asin': u'B008J3UD2U', u'price': 9.22, u'ref': u'pd_bxgy_75_2_2'},
 {u'asin': u'B008J3UD2U', u'ref': u'pd_sim_75_1'}]

what do you think about adding support for map on selector result level? So that I could do

 sel.xpath("//@data-p13n-asin-metadata").map(json.loads)

or even allow to pass list of functions

 sel.xpath("//@data-p13n-asin-metadata").map([json.loads, lambda d: d.get('asin'))

?

@Granitosaurus
Copy link

Granitosaurus commented Jun 29, 2017

I kinda like it, however it doesn't seem to be that useful, as well as map being unpythonic (list comprehensions are better! :))

sel.xpath("//@data-p13n-asin-metadata").map([json.loads, lambda d: d.get('asin'))
# vs
[json.loads(v).get('asin') for v in sel.xpath("//@data-p13n-asin-metadata").extract()]

As the comparison shows map doesn't really add much. Sure it might look a bit more tidy but list comprehension are more straigh-forward and most importantly more explicit.
Also you can use dict, set and generator comprehensions!

So I feel on the edge - would be nice but seems awfully unnecessary and inferrior to comprehensions.

@kmike
Copy link
Member

kmike commented Jun 29, 2017

I'm also on fence; let's say -0 to add such shortcut, as it doesn't add much, and it is not composable.

To make processing code shorter and more composable I'd try to explore something like pytoolz currying instead.

@redapple
Copy link
Contributor

I'm also -0 on this one. I prefer comprehension myself.

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

5 participants