Introduce latest and related packages search

[josevalim]

Now that we added package search to Hexdocs, we can start exposing it in ExDoc.
This will be done as two distinct features, latest search and related packages search.
We are also adding search feature to Hexdocs landing page and you should familiarize
yourself with the new screens there before moving forward.

Overall description

We will add new features to ExDoc that allows you to perform both search and
autocompletion on the latest or related versions of a package. We will use the
existing docs_config.js to enable this functionality. Once enabled, a dropdown
arrow will appear embedded at the end of the search allowing you to choose the
type of search you want to perform, similar to this:

The text in the search bar will indicate which mode is active:

• related - Type / to search this and related packages
• latest - Type / to search the latest version
• current - Type / to search this version

You can click the dropdown to choose the mode, these options will show up as:

• :related - Search the latest version of this package and related packages
• :latest - Search the latest version of this package
• `:current - Search the current version of this package (works offline)

To enable these modes, a searchNodes variable must be injected into docs_config.js.
This variable is an array of {name: "package-name", version: "package-version"} entries,
and it must have at least one entry, which is the current package itself and its latest
version. The modes should be available in this order and according to these conditions:

• :related - If searchNodes is available and it contains more than one entry
• :latest - If searchNodes is available and it contains more than zero entries
• :current - Always

Furthermore, we also want to improve the search results page to be closer to the one in
HexDocs and provide a more consistent experience overall. This means support for toggling
previews too.

Implementation

• Update the current search results to leverage previews and stay closer to HexDocs mockups

• Change HexDocs to include the searchNodes in all public packages today.
  We should also simplify the versionNodes one, there is no need to include "latest" on every entry
  (cc @wojtekmach)

• Add support for searchNodes in ExDoc's autocompletion and search,
  with the dropdown for configuration, using the new TypeSense API. We can use Elixir itself to test
  this feature, as its docs_config.js is generated by hand rather than Hex.pm

• Change Hex.pm to allow storing related deps and update HexDocs to emit searchNodes
  based on related deps (cc @wojtekmach)

[jimsynz]

Hey folks. I'm just trying to evaluate how much work is involved in shipping these features. I've been asked to take a look but we have a pretty constrained time budget to get it done. I'm pretty much starting from zero, so if I can get some pointers that would be awesome.

[josevalim]

Hi @jimsynz! I'd start from the first bullet item, which is to improve the look and feel of our search results, making it closer to the autocomplete results and to the results in this mockup page: hexpm/hexdocs#49 (comment) WDYT?

[zachdaniel]

I wonder if perhaps it would be better to start with the third bullet point, specifically powering autocomplete with searchNodes instead of sidebarItems. There will probably be a lot of back and forth on what the new appearance should look like in ex_doc, but ultimately swapping sidebar items for search nodes is essentially its own spec (it should all work how it did before, and sidebarItems should only ever be used to power the sidebar).

[josevalim]

There is some confusion in terms here. searchNodes is not searchData. searchNodes is a new entry in docs_config.js that tells you which packages you should query on TypeSense. Here is how it looks for req today:

var searchNodes = [{"name":"req","version":"0.4.8"}];

We don't want to load searchData for autocomplete because it can be really large in some projects. We will do changes to the search bar but the search results would look the same UI wise.

PS: we use searchNodes in some places in the codebase, i will rename them to avoid confusion.

[zachdaniel]

Interesting. So we are not looking to do cross package auto complete, only cross package search? I'm probably mixing up multiple conversations we had in the past.

[zachdaniel]

IIRC we discussed that because autocomplete is powered by a different set of data, we can't provide cross package autocomplete. FWIW I would personally value cross package autocomplete an order of magnitude higher than cross package search in terms of utility.

[josevalim]

We will do cross package autocomplete, but the cross package autocomplete comes from TypeSense APIs, not from loading searchData. Behind the scenes, searchData is what populates the TypeSense API, but from the point of view of this discussion, that's not relevant.

The goal is: see if there is a searchNodes variable, if there is, we show the dropdown in the search/autocomplete bar. Then, once we have to perform autocompletion, we send a query to TypeSense API filtering the results by the ones specified in searchNodes.

[zachdaniel]

Understood. I think the actual functionality may be better to start with personally, making the requests to the new API and populating the data etc, as that will likely be a very unambiguous "either it works or it doesn't", and then the UI can be iterated on top of the new working functionality. Just my 2 cents though :)

[josevalim]

That works for me too! I can write tomorrow steps to get these variables working.

[jimsynz]

That would be awesome thanks @josevalim

[josevalim]

If you run mix build, it will build ExDoc docs on doc/ and it now includes a docs_config.js file with the searchNodes variable, pointing to the latest published version of ExDoc.

[jimsynz]

Hi @josevalim - I'm not seeing that file after pulling the latest version:

$ rm -rf doc ; git pull origin main ; mix docs && find doc -name \*.js
From https://github.com/elixir-lang/ex_doc
 * branch              main       -> FETCH_HEAD
Already up to date.
Generating docs...
View "html" docs at "doc/index.html"
View "epub" docs at "doc/ExDoc.epub"
doc/dist/sidebar_items-9DA8DAA3.js
doc/dist/html-PJYCFMY2.js
doc/dist/search_data-B9646146.js

[zachdaniel]

I think its the elixir lang repo that has it, not ex_doc

[jimsynz]

Ahh. I just realised that @josevalim said mix build not mix docs. My bad.