RAG Retrieval Agent Not Incorporating ChromaDB Results

[michaelkawaicheung]

Description:

The notebook uses a retrieval-augmented generation (RAG) pattern with ChromaDB to answer user queries. However, the assistant does not appear to incorporate the vector-store data into its responses (e.g., it doesn’t know about “Contoso Travel” despite having documents with that information).

Potential Fixes

• Ask rag should initiate a search from the vector database using a tool or it should include the search results from ChromaDB.
• In my opinion, retrieval_agent should be used instead of assistant defined previously in the notebook.

Steps to Reproduce:

1. Initialize a ChromaDB vector store with documents containing info on “Contoso Travel.”
2. Use the following code snippet to query:

async def ask_rag(query: str, evaluator: RAGEvaluator):
    try:
        # Get response with timing
        start_time = time.time()
        response = await assistant.on_messages(
            [TextMessage(content=query, source="user")],
            cancellation_token=CancellationToken(),
        )
        processing_time = time.time() - start_time
        
        # Evaluate response
        metrics = evaluator.evaluate_response(
            query=query,
            response=response.chat_message.content,
            context=documents
        )
        
        return {
            'response': response.chat_message.content,
        }
    except Exception as e:
        print(f"Error processing query: {e}")
        return None

Sample Output

Processing Query: What luxury vacation packages does Contoso Travel offer?
Response: Contoso Travel, which typically represents a fictional or illustrative example in training materials, often provides a range of luxury vacation packages that cater to various tastes and preferences. While I don't have access to specific current offerings from a real company named Contoso Travel, I can outline some common features and types of luxury vacation packages that similar travel agencies might provide.

1. **All-Inclusive Resorts**: Packages that include accommodations, meals, drinks, and activities at high-end resorts. Popular destinations might include the Caribbean, Maldives, and Tahiti.

2. **Private Yacht Charters**: Luxury vacation packages that offer private yacht rentals, including crewed charters that can explore exotic locales while providing personalized services.

3. **Luxury Safari Packages**: These often include stays in exclusive lodges or private villas in picturesque locations like Africa, complete with guided wildlife tours and cultural experiences.

4. **Culinary or Wine Tours**: Gourmet experiences that might include private chef dinners, wine tastings in renowned vineyards, or cooking classes with local chefs in destinations famous for their cuisine.

5. **Wellness Retreats**: Packages centered around relaxation and rejuvenation, including luxury spa treatments, wellness workshops, and yoga sessions in tranquil settings.

6. **Adventure Trips**: Tailored trips that offer high-end experiences combining luxury accommodations with adventurous activities, such as guided trekking in the Himalayas or heli-skiing in Alaska.

7. **Cultural Escapades**: Exclusive tours tailored to explore the rich history and culture of regions, often including private tours of museums and historical sites, along with accommodations in luxury hotels.

8. **Honeymoon Packages**: Romantic getaways specifically designed for newlyweds, featuring beachfront bungalows, candlelit dinners, and couples' spa treatments.

When exploring luxury vacation packages with a specific agency, it's advisable to directly visit their website or contact their customer service to examine current offerings, #, and customization options according to individual preferences.

Processing Query: Can you explain Contoso's travel insurance coverage?
Response: Since Contoso is often used as a fictional example in training materials, specific details about Contoso's travel insurance coverage are not available. However, I can provide an overview of what you might typically expect from travel insurance coverage offered by a company like Contoso Travel.

### Typical Travel Insurance Coverage Components

1. **Trip Cancellation/Interruption Insurance**: 
   - This coverage provides reimbursement for non-refundable travel expenses if a trip must be canceled or interrupted due to covered reasons, such as illness, injury, a family emergency, or natural disasters.

2. **Medical Coverage**:
   - This includes expenses for medical treatment if you become ill or injured while traveling. It may also cover emergency medical transportation and evacuation, especially important for international trips.

3. **Baggage Coverage**: 
   - This component covers lost, stolen, or damaged luggage and personal belongings. It may reimburse you for the replacement value of lost items or for the costs incurred to purchase essentials if your luggage is delayed.

4. **Flight Delay/Cancellation Coverage**:
   - This reimbursement is for expenses incurred due to delays or cancellations of flights, such as accommodations, meals, and transportation.

5. **24/7 Assistance Services**:
   - Many travel insurance policies offer emergency assistance hotlines that can help with medical appointments, legal referrals, and other urgent needs while traveling.

6. **Travel Delay Coverage**:
   - Covers additional expenses incurred if your trip is delayed for reasons such as severe weather, strikes, or other unexpected events.

7. **Accidental Death and Dismemberment Insurance**:
   - This provides a benefit in case of accidental death or severe injury while traveling.

### Important Considerations

- **Exclusions**: It’s crucial to read the policy to understand what's not covered. Common exclusions include pre-existing medical conditions, risky activities (like extreme sports), or cancellations due to personal reasons not categorized as emergencies.
  
- **Deductibles**: Some policies may require you to pay a certain amount out of pocket before coverage kicks in.

- **Limits on Coverage**: There may be caps on how much you can claim for different types of expenses (like medical costs or baggage loss).

To get accurate and specific information regarding travel insurance options from Contoso Travel or any actual travel agency, I recommend checking their official website or contacting their customer service team directly. They can provide detailed policy documents and assist with any specific queries you may have.

Processing Query: What destinations and experiences are available through Contoso Travel?
Response: While Contoso Travel is a fictional example commonly used in training materials, I can provide an illustrative overview of typical destinations and experiences that a travel agency like Contoso might offer. Here are some common categories of destinations and experiences you could expect:

### Popular Destinations

1. **Tropical Islands**:
   - **Maldives**: Known for overwater bungalows, crystal-clear waters, and incredible snorkeling and diving opportunities.
   - **Bora Bora**: Famous for its picturesque lagoons and luxury resorts, perfect for honeymooners or couples.

2. **European Cities**:
   - **Paris, France**: Tours that include visits to iconic landmarks like the Eiffel Tower, Louvre, and gourmet experiences in local cafes.
   - **Rome, Italy**: Historical tours exploring ancient ruins such as the Colosseum and Vatican City, along with culinary classes.

3. **Adventure Destinations**:
   - **Costa Rica**: Packages that feature zip-lining through rainforests, surfing, and wildlife tours in national parks.
   - **New Zealand**: Experiences may include bungee jumping, hiking the Tongariro Alpine Crossing, and exploring the picturesque landscapes featured in "The Lord of the Rings."

4. **Cultural Experiences**:
   - **Kyoto, Japan**: Packages that allow travelers to visit traditional tea houses, the Arashiyama Bamboo Grove, and ancient temples.
   - **Marrakech, Morocco**: Immersive experiences exploring souks, trying traditional Moroccan cooking, and staying in luxury riads.

5. **Safari Adventures**:
   - **Kenya and Tanzania**: Luxury lodges and guided safari tours to see the Big Five (lion, leopard, rhinoceros, elephant, and Cape buffalo) in their natural habitat.

6. **Cruises**:
   - **Mediterranean Cruises**: Itineraries that include stops in multiple countries, such as Italy, Greece, and Spain, with opportunities for excursions in each port.
   - **Alaskan Cruises**: Experiences focused on wildlife viewing, glacier touring, and exploring quaint coastal towns.

7. **Wellness Retreats**:
   - **Bali, Indonesia**: Packages that include yoga retreats, spa treatments, and holistic wellness programs in tranquil settings.
   - **Sedona, Arizona**: Experiences centered around healing, hiking, and spiritual wellness in a stunning natural environment.

### Unique Experiences

1. **Culinary Tours**: A focus on local food experiences, including cooking classes, wine tastings, and market visits to explore regional cuisines.

2. **Wildlife Conservation Programs**: Opportunities to participate in volunteer programs that support wildlife conservation efforts, such as working with elephants in Thailand or rescuing sea turtles.

3. **Luxury Train Journeys**: Experiences like traveling on the Belmond Orient Express or Rocky Mountaineer, combining scenic vistas with sumptuous dining.

4. **Personalized Itineraries**: Customizable travel packages allowing clients to select their preferred activities, accommodations, and transportation options.

5. **Digital Nomad Packages**: Tailored itineraries for remote workers combining travel with work facilities, including co-working spaces in popular locations.

For specific offerings, experiences, and destinations from Contoso Travel or any other real-world travel agency, it’s best to check their official website or contact them directly to explore current packages and promotional deals.

[github-actions]

👋 Thanks for contributing @michaelkawaicheung! We will review the issue and get back to you soon.

[ShivamGoyal03]

Hi, @michaelkawaicheung Thanks for pointing out, I had checked using retrieval_agent but it still gives me output. Maybe its due to the model - gpt-4o-mini, that we're using in our Azure ChatCompletion model.

Did you found any other potential fix?

---

Edited:

I have found the issue and fixed it in the latest PR

[ShivamGoyal03]

Hi, @koreyspace @leestott @michaelkawaicheung , this issue is fixed in ae38427

Please let me know if there are any other issues. If not, we can go ahead and close this.