[Enhancement]: get doc ids by batch

[SpadeA-Tang]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Milvus row id does not match with tantivy doc id, so we first get the tantivy doc id produced by query,
then we search the related doc id, which stores in the "doc_id" field with type fast_field.

This one by one process is not efficient: 1. dynamic dispatch is involed when getting doc id in "doc_id" field, 2. we cannot benefit from SIMD optimization. 3. memory locality and so on.

Why is this needed?

No response

Anything else?

No response

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Milvus row id does not match with tantivy doc id, so we first get the tantivy doc id produced by query, then we search the related doc id, which stores in the "doc_id" field with type fast_field.

This one by one process is not efficient: 1. dynamic dispatch is involed when getting doc id in "doc_id" field, 2. we cannot benefit from SIMD optimization. 3. memory locality and so on.

Why is this needed?

No response

Anything else?

No response

is it a good idea to have a specialized map implementation for doc id to accelerate

[xiaofan-luan]

instead of optimizing filter, we need a data structure really good at retrieve, like a hash map

[SpadeA-Tang]

#40608 optimizes it by utilizing tantivy's internal batch API and can achieve better performance. @xiaofan-luan