Support for Embeddings

[brainlid]

Here's a place to start talking about how embeddings can be supported. It was started as issue #40, but it isn't concrete enough to do anything with yet.

[cpursley]

Here's a neat talk on the subject: https://www.youtube.com/watch?v=RFnZB76KVWk

[cpursley]

Also worth pointing out:

• https://github.com/pgvector/pgvector-elixir
• https://github.com/toranb/pgvector-search
• https://github.com/dvcrn/chatgpt-ui (uses tokenizers lib)

[slashmili]

I'd like to suggest these video series from LanChain team: RAG From Scratch

It touches RAG in general and also embedding.

Using Embedding has two side of story:

1. Indexing: Which I think could be consider separately as there are so many techniques on how to do it.
2. Querying: Which we basically just need to know which model to use to do the embedding of the query and some information on how to query an external db.

Something neat about how the python version of LangChain works on handling template and passing function around, e.g: (full source code)

template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

...
retriever = vectorstore.as_retriever()
...
rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

rag_chain.invoke("What is Task Decomposition?")

As you see it allows that the context inside the template to be a string or a function in this case and is possible to be evaluated when the final text is created

[cpursley]

I've been experimenting with pgvector and Bumblebee / NX for generating embeddings with open models. Pretty impressive results with my initial testing. Now I'm thinking how this could work in a langchain type of setup where the chain could know when make a retrieval query (similar to how tools/functions work) and pull the retrieved data into the context.

[brainlid]

Awesome @cpursley! That's so cool!

I recently went to Google Cloud Next 2024 and a presenter said that the next big thing with RAG was making a RAG tool available to an Agent to selectively use. That way an Agent might do an internet search, a relational DB lookup, or a vector DB document lookup. It's a hybrid approach.

You're thinking in the same line.

We might want to create a tool definition for a RAG interface. Then then LLM could choose to query that tool for the information.

On a related note, it became clear at the conference that a user's query may need to be pre-processed and restated in a way that will search better against the vector data store.

For instance, if the documents being searched are legal documents or perhaps technical documents, the user's question may be in non-legal or non-technical terms and would not have any good hits. So restating the user's question in legal or technical terms before querying the vector store will yield better results.

Because of this, that pre-processing step might happen as a shorter, separate chain. That helps keep the token counts low and responses faster.

[cpursley]

It did occur to me that having the LLM re-write/pre-process the user query might result in better results. This might be a chicken-or-egg situation (is the LLM smart enough to convert laymans terms to effective technical terms, etc).

The other tricky part is for the chain/LLM combo to actually understand when it should call the RAG tool. I kind of got that working via a function that queries the RAG but wondering if there's a better way. Not sure how others are handling that.

[slashmili]

coincidentally the people behind LangChain(Py) dropped a video about Adaptive RAG https://www.youtube.com/watch?v=-ROS6gfYIts

[cpursley]

The retrieval grader stuff was really interesting and answers a question I've had about handling the quality of retrieved results. Good find @slashmili!