Unexpected index size after inserting elements with same ids with DirectMap.Hashtable

[javierjuan]

Summary

I am using an IVF index with DirectMap.Hashtable to allow adding elements with arbitrary ids. After training the index, and adding 10000 elements (with a specific set of ids), checking the index size through index.ntotal correctly returns 10000 . However, if I add again the same set of elements with the same ids the index size increases to 20000.

I don't know if I am missing something but what I would expect here is that if I add elements with ids that are currently in use in the index, those elements in the index will be replaced with the new ones. However, when I check the index.ntotal it returns 20000, like if the index had grown with 10000 new elements. But in addition, if I try to reconstruct the elements ranging from 10000 to 19999, a key not found exception is raised (as expected) because these ids do not exist.

Am I missing something? Is it not the correct way to check the index size by querying index.ntotal?

Thanks

Platform

OS: Windows 10

Faiss version: 1.7.2

Installed from: faiss-cpu from pip

Faiss compilation options: None

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

n = 10000
d = 384
x = np.random.randn(n, d).astype(np.float32)
ids = np.arange(n, dtype=np.int64)

index = faiss.index_factory(384, f'IVF{int(2 * math.sqrt(n))},Flat', faiss.METRIC_INNER_PRODUCT)
index.set_direct_map_type(faiss.DirectMap.Hashtable)
index.train(x)

print(index.ntotal)  # 0

index.add_with_ids(x, ids)
print(index.ntotal)  # 10000

index.add_with_ids(x, ids) # what I expected here is to replace the ids with the new x
print(index.ntotal)  # 20000 ??

for i in range(2*n):
    try:
        index.reconstruct(i)
    except Exception as error:
        print(f'Error reconstructing key {i}')  # from i=10000 to 19999 throw key not found exceptions

[mdouze]

This is expected. The index is perfectly fine with having several vectors with the same ids (they could be class labels for example).
The direct map is used only for reconstruction. If several vectors have the same id, then an arbitrary one will be returned by reconstruct(vector_id).

[javierjuan]

Sorry, I did not know this was the expected behavior. So, is there an index (or configuration) that allows only one vector per id, and updates the "old" vector if another one is added with an id that is currently in use?

Moreover, right now I have a new doubt. You said that "The direct map is used only for reconstruction" but if I want to add and remove elements with non-sequential ids, even if I never reconstruct them, I still need DirectMap.Hashtable, is it correct?

Thanks

[czpmango]

@javierjuan u can use update_vectors instead of add_with_ids