Is Result object cacheable?

[Alkalit]

I'm trying to save Response object in redis cache in django project

from django.core.cache import cache

query = \
    {
      "query": {
# some query here
      }
    }

searcher = Search(using=haystack_connections['default'].get_backend().conn)
searcher = searcher.update_from_dict(query)[:10]

results = searcher.execute()
cache.set('foo', result, timeout=60*60)
cache.get('foo')

/home/gnom/www/env/apps/elasticsearch_dsl2/result.pyc in __repr__(self)
     24 
     25     def __repr__(self):
---> 26         return '<Response: %r>' % self.hits
     27 
     28     def __len__(self):

/home/gnom/www/env/apps/elasticsearch_dsl2/result.pyc in hits(self)
     47             except AttributeError as e:
     48                 # avoid raising AttributeError since it will be hidden by the property
---> 49                 raise TypeError("Could not parse hits.", e)
     50 
     51             # avoid assigning _hits into self._d_

TypeError: ('Could not parse hits.', AttributeError(u"'Response' object has no attribute '_callbacks'",))

or:

cache.set('foo', list(result), timeout=60*60)
cache.get('foo')


/home/gnom/www/env/apps/elasticsearch_dsl2/result.pyc in <genexpr>((key,))
     83         return '<Result(%s): %s>' % (
     84             '/'.join(getattr(self.meta, key) for key in
---> 85                       ('index', 'doc_type', 'id') if key in self.meta),
     86             super(Result, self).__repr__()
     87         )

/home/gnom/www/env/apps/elasticsearch_dsl2/utils.pyc in __getattr__(self, attr_name)
    121         except KeyError:
    122             raise AttributeError(
--> 123                 '%r object has no attribute %r' % (self.__class__.__name__, attr_name))
    124 
    125     def __delattr__(self, attr_name):

AttributeError: 'Result' object has no attribute 'meta'

dsl lib version is (2, 2, 0)

[honzakral]

I added some tests for pickleability of Response and Hit classes and everything seem to work fine, could you please test the maste version of the library with your code?

Thank you!

[Alkalit]

I tried to use master branch - it fails again.

My session from stage server:

In [1]: from app.views import get_results

In [2]: from django.core.cache import cache

In [3]: results = get_results()
[17/Jan/2017 10:00:23] INFO [elasticsearch.trace:73] curl -XGET 'http://localhost:9200/_search?pretty' -d '{
  "from": 0,
  "query": {
    "function_score": {
      "filter": {
        "bool": {
          "must": [
            {
              "term": {
                "anchor_show_up": true
              }
            },
            {
              "terms": {
                "link": [
                  "/moskva/sell_syire_vtorichka_granula_polietilen/",
                  "/moskva/sell_syire_vtorichka_granula_pnd/",
                  # ... about 4k rows
                ]
              }
            },
            {
              "term": {
                "django_ct": "app.seolink"
              }
            }
          ],
          "must_not": [
            {
              "terms": {
                "anchor_name": []
              }
            }
          ]
        }
      },
      "functions": [
        {
          "random_score": {
            "seed": "fa003133eb"
          }
        }
      ]
    }
  },
  "size": 10
}'

In [4]: cache.set('foo', results, timeout=60*60)
Out[4]: True

In [5]: cache.get('foo')
Out[5]: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/user/www/.env/local/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
    381                             if callable(meth):
    382                                 return meth(obj, self, cycle)
--> 383             return _default_pprint(obj, self, cycle)
    384         finally:
    385             self.end_group()

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
    501     if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
    502         # A user-provided repr. Find newlines and replace them with p.break_()
--> 503         _repr_pprint(obj, p, cycle)
    504         return
    505     p.begin_group(1, '<')

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _repr_pprint(obj, p, cycle)
    692     """A pprint that just redirects to the normal repr function."""
    693     # Find newlines and replace them with p.break_()
--> 694     output = repr(obj)
    695     for idx,output_line in enumerate(output.splitlines()):
    696         if idx:

/home/user/www/project/apps/elasticsearch_dsl2/response/__init__.py in __repr__(self)
     26 
     27     def __repr__(self):
---> 28         return '<Response: %r>' % self.hits
     29 
     30     def __len__(self):

/home/user/www/project/apps/elasticsearch_dsl2/response/__init__.py in hits(self)
     51             except AttributeError as e:
     52                 # avoid raising AttributeError since it will be hidden by the property
---> 53                 raise TypeError("Could not parse hits.", e)
     54 
     55             # avoid assigning _hits into self._d_

TypeError: ('Could not parse hits.', AttributeError(u"'Response' object has no attribute '_search'",))

In [7]: import elasticsearch_dsl2

In [8]: elasticsearch_dsl2.VERSION
Out[8]: (5, 1, 0)

In [9]: cache.set('foo', list(results), timeout=60*60)
Out[9]: True

In [10]: cache.get('foo')
Out[10]: ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/user/www/.env/local/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
    366                 if cls in self.type_pprinters:
    367                     # printer registered in self.type_pprinters
--> 368                     return self.type_pprinters[cls](obj, self, cycle)
    369                 else:
    370                     # deferred printer

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in inner(obj, p, cycle)
    550                 p.text(',')
    551                 p.breakable()
--> 552             p.pretty(x)
    553         if len(obj) == 1 and type(obj) is tuple:
    554             # Special case for 1-item tuples.

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
    381                             if callable(meth):
    382                                 return meth(obj, self, cycle)
--> 383             return _default_pprint(obj, self, cycle)
    384         finally:
    385             self.end_group()

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
    501     if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
    502         # A user-provided repr. Find newlines and replace them with p.break_()
--> 503         _repr_pprint(obj, p, cycle)
    504         return
    505     p.begin_group(1, '<')

/home/user/www/.env/local/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _repr_pprint(obj, p, cycle)
    692     """A pprint that just redirects to the normal repr function."""
    693     # Find newlines and replace them with p.break_()
--> 694     output = repr(obj)
    695     for idx,output_line in enumerate(output.splitlines()):
    696         if idx:

/home/user/www/project/apps/elasticsearch_dsl2/response/hit.py in __repr__(self)
     31             '/'.join(
     32                 getattr(self.meta, key)
---> 33                 for key in ('index', 'doc_type', 'id')
     34                 if key in self.meta),
     35             super(Hit, self).__repr__()

/home/user/www/project/apps/elasticsearch_dsl2/response/hit.py in <genexpr>((key,))
     32                 getattr(self.meta, key)
     33                 for key in ('index', 'doc_type', 'id')
---> 34                 if key in self.meta),
     35             super(Hit, self).__repr__()
     36         )

/home/user/www/project/apps/elasticsearch_dsl2/utils.py in __getattr__(self, attr_name)
    113         except KeyError:
    114             raise AttributeError(
--> 115                 '%r object has no attribute %r' % (self.__class__.__name__, attr_name))
    116 
    117     def __delattr__(self, attr_name):

AttributeError: 'Hit' object has no attribute 'meta'

In [11]: import elasticsearch

In [12]: elasticsearch.VERSION
Out[12]: (2, 1, 0)

In [13]: from django.conf import settings

In [14]: print settings.CACHES
{'default': {'OPTIONS': {'CLIENT_CLASS': 'django_redis.client.DefaultClient'}, 'LOCATION': 'redis://127.0.0.1:6379/1', 'BACKEND': 'django_redis.cache.RedisCache'}}

[amosjyng]

I am getting the same error. I write to a pickled file using:

import pickle

result = some_es_search.execute()
with open('test.pickle', 'wb') as pickle_file:
    pickle.dump(result, pickle_file)

and then read from it again using:

import pickle

with open('test.pickle', 'rb') as pickle_file:
    result = pickle.load(pickle_file)

When I try to print the results:

print(result.hits)

I am hit with the following error:

TypeError: ('Could not parse hits.', AttributeError("'Response' object has no attribute '_search'",))

This is on master

[amosjyng]

The problem is that when the pickled Response object is read back, it is missing _search and _hits. Only _d_ is there.

This is because of this line.

[amosjyng]

It works a little more if I try to pickle _hits as well like here. But then I run into things like AttributeError: 'Hit' object has no attribute 'meta'

[honzakral]

Thank you for the report, we changed the inner workings of Result and forgot to update its serialization, I will look into it.

[Alkalit]

@honzakral Can you please add this fix also to 2.X branch?

[honzakral]

@Alkalit done as a455fdd

thanks for the reminder!

[amosjyng]

Thanks guys!