Why does LanceDB throw when doing a full text search with limit(2) but not with limit(1)?

[charnould]

Hello,

First, thanks for making LanceDB!
I'm new to Lance and I have a question/problem.

I've added some data into a Lance DB and I'm toying with full text search (with Python).

This code works (= it returns a unique result)

working =  tbl.search("loyer").limit(1).select(["lemma"]).to_list()
print(working)

But this one doesn't:

not_working  =  tbl.search("loyer").limit(2).select(["lemma"]).to_list()
print(not_working)

Difference between those: limit(1) vs. limit(2) or 3, 4, 5...

It throws:

Traceback (most recent call last):
  File "lance.py", line 11, in <module>
    test = tbl.search("loyer").limit(3).select(["lemma"]).to_pandas()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lancedb/query.py", line 195, in to_pandas
    return self.to_arrow().to_pandas()
           ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lancedb/query.py", line 463, in to_arrow
    output_tbl = self._table.to_lance().take(row_ids, columns=self._columns)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lance/dataset.py", line 452, in take
    return pa.Table.from_batches([self._ds.take(indices, columns)])
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: LanceError(Arrow): Invalid argument error: Last offset 651 of LargeUtf8 is larger than values length 217, /Users/runner/work/lance/lance/rust/lance-core/src/encodings/binary.rs:397:26

Do you have any clue?
Thanks a lot

[wjones127]

Hi @charnould. That seems like a bug. Could you share how you created your table, added data, and created the index? Then we can find steps to reproduce.