Serve results as an API

[mporracindie]

It would be really useful if we could mount this on a machine (or multiple) and allow the search to be done through an API.
This could allow for multiple use cases like building a frontend for the tool.

[rsyi]

This is a super cool idea, but I haven't built an API before in rust, so this could take quite some time (or might need some external help on this).

In the meantime, a roundabout way to do this could be to host this on github, then use the github search APIs to search within the repo. It's not quite as excellent of a search experience as the rust interface, but it could do.

Related to #82.

[mporracindie]

I have never coded rust but I would like to at least give it a go.

I'm still reading and getting a full understanding of how whale works under the hood.
All the ETL processes are in python and the CLI is in rust and as far as I can understand the metadata search is being done in rust too but I'm not being able to find where. Can you point me in the right direction?

[rsyi]

Love it! @mporracindie

Yeah the entire CLI interface is being done in rust, in cli/. I'll post a high-level overview for you to get started. :)

• main.rs contains all the CLI bindings. This line is the line that invokes the search. It calls the Whale::run_with function, which I'll explain next...
• lib.rs contains the code that is referenced and executed by main.rs. Notably, there's a Whale struct that houses all of the high-level functionality of the whale cli. l.73 is where the run_with function is defined. Notably, it calls skimmer::table_skim, which I'll explain next.
• skimmer.rs contains code that actually runs the underlying library to conduct the search. We use a library called skim, which renders the GUI (check out the repo here). We feed in a plaintext file (the "manifest") to make sure the GUI renders quickly even at high volumes. Skim returns the selected results, then we provide some custom logic to open the selected files.

I think the key here is actually just to figure out how to either (a) open an API using the skim library, or (b) figure out a way to eagerly results from the search engine given a query. It may make sense to open an issue on lotabout's repo and see if he has any suggestions. :)

Honestly, though, I imagine this might be quite a tricky project to tackle. It may make sense to join our slack channel to get more direct input from the wider community :)

[mporracindie]

Just join the channel so we can maybe continue the conversation there but just so anyone else can read it I will post it here too.

From this issue in skim we know that what is being used to match is actually fuzzy-matcher.
It's also interesting that the original implementations are in C++ and Javascript the last one being quite optional for building a simple API.

I'll keep investigating and see what I can find/do.

PS: just as a curiosity this seems to be the algorithm our loved sublime use to have in the past. (article)