A basic implementation of a LangChain chatbot app operated from terminal.

This is a demo project to show how Aim enables building an observability and automation layer for your entire AI System.

About • Using the Chatbot • Observe and Automate with Aim •

ℹ️ About

Repo Structure

This repo contains two python packages:

• chatbot: the chatbot implementation with LangChain
• chatbot_logger: end-to-end custom observability package for chatbot built with Aim framework.

About Aim

Aim is an open-source development framework for building AI Systems observability and automation.

Aim captures the relationships between different components of the System and enables these three capabilities:

• Define and Log AI metadata from everywhere in AI Infra into centralized storage
• Build low-code, composable custom observability for logged metadata
• Build automations for logged metadata.

With this, Aim enables a new artifact - custom observability and automation layer for every AI System that captures all relevant relationships to fully show the System lineage, its evolution, its state as well as zoom-in zoom-out into every component of it.

Aim helps to recover the broken development lifecycle in AI Systems.

Aim is python-composable. Aim building blocks can be packaged as python packages and reused across different projects. Both separately and all together. See the package chatbot_logger in this repo.

Aim's mission is to democratize AI Dev tools to recover the broken AI Systems development lifecycle.

Using the Chatbot

Installation

1. Clone the repo
2. Add .env file at chatbot with serpapi_key and openai_key keys
3. Install the logger pip install -e ./chatbot_logger (in editable mode)
4. Install the chatbot pip install -e ./chatbot (in editable mode)

Usage and development

There are two modes of chatbot iteration: Dev and Production.

Production

Execute cd chatbot && chatbot run to run the chatbot. Each chatbot run is a "production" run and uses the version number from chatbot/VERSION as its release version.

Dev

Execute cd chatbot && chatbot run --dev to run the chatbot

Observe and Automate with Aim

The chatbot_logger built with Aim tracks the chatbot both across development and production.

Using Aim

All commands run in dir langchain-chatbot

• Start Aim server: aim server --package chatbot_logger
  • Note: this command to be automatically ran with aim up in future iterations
• Start Aim UI: aim up --package chatbot_logger
• Checkout Aim docs: aim up --port 43001 --package docs

The chatbot development toolkit

• Chatbot System Lineage page: full overview of the development and releases of the chatbot

• Production Sessions page: full overview of the production sessions by the users

• User Analytics page: select user and see the users' activity breakdown.

• Production Session: observe the User sessions on the chatbot in detail

Using the built-in playground documentation

Aim comes with a built-in playgroun d documentation. Here is how to start it: aim up --port 43001 --package docs This command will start an Aim instance that's also a playground documentation to experiment with and customize your toolkit.

Next steps: make it your own

There are many ways this package can be extended

• Add new metrics that track chatbot efficiencies
• Add aggregate metrics of the openai cost
• potentially restructure the whole package to better see the lineage of the experiment -> dev session -> release -> user session -> issue -> experiment ->...