Jetson Copilot is a reference application for a local AI assistant.
It demonstrates two things;
- Running open-source LLMs (large language model) on device
- Augmenting the LLM to have access to your locally indexed knowledge (RAG, retrieval-augmented generation)
Important
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
| OSS | Repo URL | Usage |
|---|---|---|
| Ollama | GitHub | To host and run LLMs locally, including embedding models for building index from documents |
| LlamaIndex | GitHub | Data framework for LLM, used mainly to realize RAG pipeline. |
| Streamlit | GitHub | Python library to create an interactive web app |
If this is your first time to run Jetson Copilot on your Jetson, first run setup_environment.sh to ensure you have all the necessary software installed and the environment set up.
git clone https://github.com/NVIDIA-AI-IOT/jetson-copilot/
cd jetson-copilot
./setup_environment.shIt will install the following, if not yet.
- Chromium web browser
- Docker
Important
It also adds the current user into docker group using newgrp command.
Check to see if you can just issue the following command (without using sudo).
docker psIf you get permission denied error, exit from the session (by executing exit command), reboot if possible (so that it will take effect in any sessions), and log/ssh back in.
You should now be able to issue docker ps.
cd jetson-copilot
./launch_jetson_copilot.shlaunch_cui.mp4
This will start a Docker container and start a Ollama server and Streamlit app inside the container. It will shows the URL on the console in order to access the web app hosted on your Jetson.
With your web browser on Jetson, open the Local URL (localhost). Or on a PC connected on the same network as on your Jetson, access the Network URL.
Local URL: http://localhost:8501
Network URL: http://10.110.50.252:8501 Note
You will need the Internet connection on Jetson when it launches for the first time, as it will pull the container image (and download the default LLM and embedding model when web UI starts for the first time).
When you access the web UI for the first time, it will download the default LLM (llama3) and the embedding model (mxbai-embed-large).
Tip
If you are on Ubuntu Desktop, a frameless Chromium window will pop up to access the web app, to make it look like an independent application. You need to close the window as stopping the container on the console won't shutdown Chromium.
desktop_launch.mp4
llm_question_OS-and-version.mp4
You can use Jetson Copilot just to interact with a LLM without enabling RAG feature.
By default, Llama3 (8b) model is downloaded when running for the first time and use as the default LLM.
You will be surprized how much a model like Llama3 is capable, but may soon find limitations as it does not have information prior to its cutoff date nor know anything about your specific subject matter.
rag_usb-device-mode.mp4
On the side panel, you can toggle "Use RAG" on to enable RAG pipeline.
The LLM will have an access to a custom knowledge/index that is selected under "Index".
As a sample, a pre-build index "_L4T_README" is provided.
This is built on all the README text files that supplied in the "L4T-README" folder on the Jetson desktop.
It is mounted as
/media/<USER_NAME>/L4T-README/once you executeudisksctl mount -b /dev/disk/by-label/L4T-README.
You can ask questions like:
What IP address does Jetson gets assigned when connected to a PC via a USB cable in USB Device Mode?
rag_build_index.mp4
You can build your own index based on your local and/or online documents.
First, on the console (or on the desktop) create a directory under Documents directory to store your documents.
cd jetson-copilot
mkdir Documents/Jetson-Orin-Nano
cd Documents/Jetson-Orin-Nano
wget https://developer.nvidia.com/downloads/assets/embedded/secure/jetson/orin_nano/docs/jetson_orin_nano_devkit_carrier_board_specification_sp.pdfNow back on the web UI, open the side bar, toggle on "Use RAG", then click on "➕Build a new index" to jump to a "Build Index" page.
Give a name for the Index you are to build. (e.g. "JON Carrier Board")
Type in the field and hit Enter key, then it will check and show what path will be created for your index.
And then from the drop select box under "Local documents", select the directory you created and saved your documents in. (e.g. /opt/jetson_copilot/Documents/Jetson-Orin-Nano).
It will show the summary of files found in the selected directory.
If you want to rather only or additionally supply URLs for the online docuemnts to be ingested, fill the text area with one URL per a line.
You can skip this if you are building your index only based on your local documents.
Note
On the sidebar, make sure mxbai-embed-large is selected for the embedding model.
Use of OpenAI embedding models is not well supported and needs more testing.
Finally, hit "Build Index" button.
It will show the progress in the drop-down "status container", so you can check the status by clicking on it.
Once done, it will show the summary of your index and time it took.
You can go back to the home screen to now select the index you just built.
TODO
Streamlit based web app is very easy to develop.
On web UI, at the top-right of the screen, choose "Always rerun" to automatically update your app every time you change the source codes.
See Streamlit Documentation for the detail.
In case you make more fundamental changes, you can also manually run streamlit app.
cd jetson-copilot
./launch_dev.shOnce in container;
streamlit run app.pylaunch_dev.mp4
└── jetson-copilot
├── launch_jetson_copilot.sh
├── setup_environment.sh
├── Documents
│ └── your_abc_docs
├── Indexes
│ ├── _L4T_README
│ └── your_abc_index
├── logs
│ ├── container.log
│ └── ollama.log
├── ollama_models
└── Streamlit_app
├── app.py
├── build_index.py
└── download_model.py
Following directories inside the jetson-copilot directory are mounted in the Docker container.
| Directory Name | Description |
|---|---|
Documents |
Directory to store your documents to be indexed |
Indexes |
Directory to store pre-built (or built-by-you) indexes for LLM to perform RAG on |
logs |
Directory for the app to store log files |
ollama_models |
Directory for the ollama server to store download models |
stremlit_app |
Directory for Python scripts to make up the web app |
If you find any issue, please check GitHub Issues of the Jetson Copilot repo and file an issue there.
If you get "permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock" issue, your current user account in Linux may not be added to the group docker and/or the change is not taking effect yet.
So issue the following commands to ensure you add your user account in the group, and exit from the current session, reboot if possible (so that it will take effect in any sessions), then log/ssh back in.
sudo systemctl restart docker
sudo usermod -aG docker $USER
newgrp docker
exitYou can check if you succeed by issuing docker ps.
If you run launch_jetson_copilot.sh and get "unable to setup input stream: unable to set IO streams as raw terminal: input/output error", then you can just use the launch_dev.sh and manually start the streamlit app.
./launch_dev.shAnd once in the container;
streamlit run app.pyYou can use Jetson Copilot in multiple ways.
Users can easily run an LLM on Jetson without relying on any cloud services.
They may find the AI assistance on some tasks useful, like to find out the right command to use on Linux system. They can even expand the LLM knowledge by building the local index based on their own documents that LLM can access.
Developers can use Jetson Copilot as a reference for building their own AI assistant that possesses some specific domain area's or product's knowledge.
Please see LICENSE file.
Pushed to public, still partially in development.
- LLM download UI (
download_model.py) - Use
faissfor Vector DB - Support OpenAI embedding models