Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Why does my overview() load forever? #71

Closed
gitgithan opened this issue Feb 19, 2023 · 4 comments
Closed

Why does my overview() load forever? #71

gitgithan opened this issue Feb 19, 2023 · 4 comments

Comments

@gitgithan
Copy link

In colab, visualizing both hnsw and faiss using my own index with overview or searchRandTestVec shows the cell has finished running but the blue spinning circle goes on forever on a white output.

The same notebook can show the examples from https://colab.research.google.com/drive/12L_oJPR-yFDlORpPondsqGNTPVsSsUwi#scrollTo=N3qqBAYxYcbt though.

Is the problem with the index used?
I tried reducing the dimensionality to 5, even lower than the 10 in the example but it still loads forever.

How to investigate how the indexes at

hnswSource = 'hnswlib'
hnswIndexFile = 'https://assets.zilliz.com/hnswlib_hnsw_voc_17k_1f1dfd63a9.index'

ivfflatSource = 'faiss'
ivfflatIndexFile = 'https://assets.zilliz.com/faiss_ivf_flat_voc_17k_ab112eec72.index'

were built?

I tried to find the data using

import requests

hnswIndexFile = 'https://assets.zilliz.com/hnswlib_hnsw_voc_17k_1f1dfd63a9.index'
localFilename = 'hnswlib_hnsw_voc_17k_1f1dfd63a9.index'

response = requests.get(hnswIndexFile)
with open(localFilename, 'wb') as f:
    f.write(response.content)

p = hnswlib.Index(space='l2',dim=10)
p.load_index(localFilename)
p.get_items()

but got empty list.
I was expecting get_items() to show the data added to the index

@alwayslove2013
Copy link
Collaborator

federPy = FederPy(hnswIndexFile, hnswSource, **hnswViewParams)

Note that the index file required by federPy cannot be a local file. You can use python http.server to build a simple http server.

The reason for this is that the actual output of federPy is an html file like that, which goes through the browser to fetch the index file and render it using javascript. There are many security restrictions on browsers fetching local files directly.

In addition, place the index file on the web in order to facilitate the sharing of visualizations created by feder.

@gitgithan
Copy link
Author

gitgithan commented Feb 22, 2023

Any publicly accessible url is fine right?
I upload mine to s3 with public access but both hnsw/faiss still shows blank output

https://federindexes.s3.ap-southeast-1.amazonaws.com/faiss.index
https://federindexes.s3.ap-southeast-1.amazonaws.com/hnsw.index

If there were problems with how the index was built, will it also show blank?
Not sure if it's index access or additionally index content issue

@alwayslove2013
Copy link
Collaborator

It seem that your s3 file does not allow cross-domain access.

@gitgithan
Copy link
Author

gitgithan commented Feb 23, 2023

Thanks looks like the visualizations are showing after allowing all origins in CORS policy.
Images are showing too if they are on public s3

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants