How could we enhance a foundation model with additional new local data, specified at runtime?

[classicjazz]

The foundation model (e.g. LLaMA) is limited to those sources scraped by the model's publisher, prior to the creation of the model. I would like to augment a given model's "knowledge" by manually specifying at run time a directory containing "authoritative" data files (e.g. HTML, TXT, PDF, XLS, XML) that are then incorporated into the response from the AI when presented with an instruction. By authoritative, I mean that, to the extent that data in the specified directory conflicts with pre-existing data in the model, it is given greater weight/importance.

For example, a new product is released or a news event occurs. I manually save 50 relevant web pages, text files, etc. about that product or news event to a given local directory. I want to then run llama.cpp, which outputs a news article, abstract, key points, essay, blog article, poem, song, etc. based on the new data.

I usually run a Bash script that loads llama.cpp with desired paramters. I am not sure whether llama.cpp itself loads and processes data in the directory or whether there is a prior step to somehow modify/add a layer to the model, which is then invoked by llama.cpp.

Note: for this use case, I am envisioning that it is ad hoc and limited to local directories for security/privacy/limited long term reuse. It should be capable of processing the new data within a reasonable amount of time on consumer grade hardware (e.g. Macbook or Mac Studio).

Thanks

[j-f1]

That sounds like a big challenge — especially the part about getting it to treat that info as authoritative. You’d be significantly advancing LLMs by making something like that work well locally.

[BetaDoggo]

Currently the best way to do this would probably be to train a LoRA for the new data that you want the model to understand. This is not fast but it should be possible on consumer hardware(24gb of vram or less). This probably isn't what you're looking for though.

[classicjazz]

This is along the same lines that I was thinking, applied as a layer, likely as a separate step in the Bash script prior to invoking llama.cpp.

I probably am not a typical user but I have an Apple Studio Max with 64GB of unified memory as my primary workstation.

[Technetium1]

https://github.com/lastmile-ai/llama-retrieval-plugin

[classicjazz]

Thanks for the link. I think the llama retrieval plugin use case is similar to what I suggested here:

#465

[Technetium1]

Yes 😄

[FrankDMartinez]

Wouldn’t this simply import data limitations found in ChatGPT? For example, suppose ChatGPT’s dataset deliberately excludes information on topic X; doesn’t this mean Llama won’t import it as well?

[Technetium1]

@FrankDMartinez no. Even so, if you are asking for things explicitly disallowed, you can prompt engineer your way around.