Add Support for Image Return in Agent Tools

[gabriansa]

Describe the feature

I would like to propose adding support for tools that can return images. This would enable agents to generate, process, and analyze visual data as part of their workflow.

Use Cases:

1. Data Visualization

   • Agents could generate plots and charts from numerical data
   • Analyze trends and patterns visually
   • Make data-driven decisions based on visual representations

2. Image Processing

   • Generate or modify images based on specifications
   • Create diagrams or flowcharts
   • Process and return modified images

Dummy Example Scenario:

@function_tool
def create_plot(data: list[float]) -> Image:  # New Image return type
    """Creates a line plot from the given data and returns it as an image."""
    plt.figure()
    plt.plot(data)
    # Convert plot to image
    return plot_to_image()

[rm-openai]

We should definitely do this at some point. Main blocker is that the OpenAI API doesn't support this yet.

[gabriansa]

What about as an additional message after the tool call?

Dummy Example

messages.append({
    "role": "assistant",
    "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{base64_image}",
            },
        },
    ],
})

[rm-openai]

Yeah that would be cool. Would require a somewhat large refactor to the tools pipeline, to allow returning Content from a tool call instead of just str

[weathon]

Yes, I would like this to be added as well

[kkilchrist]

Edit - I'm an idiot and while the solution appeared to be working, the agent was actually just filling in context extremely well. The images were not being accessed.

My thought process was that it may be possible to add the image to the conversation by its ID on the OpenAI.

I think I will end up with a solution like @Eslsamu describes below.

[gabriansa]

I like this. However, this does not really make a tool return an image. Instead it seems like you are just inputting an image/s before the start of the agent's run. While the agent is running it still won't have access to images that are returned by tools.

Am I missing something?

[Eslsamu]

I ended up with a similar solution. I am running an Agent within a tool, and attach the images and some context to the input and returning an evaluation/description of the images. It's not ideal as it needs some prompt engineering to make process appear seamless to the client. The fact that the "user" is attaching the images to the history is often a problem. Another is to really make it clear to the image-analysis agent-tool what the purpose of the image inspection is, and how the response need to be crafted to give the relevant image description to the main agent.

Letting tools return images would still be much better.

Edit:

Attaching the solution for reference. It's about inspecting results from a filesearch of a product database visually.

from pydantic import BaseModel

"""
Module for reviewing product previews using an AI agent.
"""

class Review(BaseModel):
    "A single review. Max 1-2 sentences."

    review: str


class ProductSearchResultsReview(BaseModel):
    "A list of product reviews"

    reviews: list[Review]


review_agent = Agent(
    name="Product Review Agent",
    instructions=(
        "TASK: Review the product previews based on the conversation context. "
        "GOAL: Filtering & Evaluating candidate product from a database search. "
        "Note: Multiple candidates can be good fits."
    ),
    model="gpt-4.1-mini",
    output_type=ProductSearchResultsReview,
)


@function_tool
async def review_product_preview(filenames: list[str], context: str) -> ProductSearchResultsReview:
    """Delegate to another VLM agent to inspect the product preview of one or multiple file. Provide all necessary context in 1-5 sentences, e.g. 'User is searching for a pair of black leather gothic boots with subtle distressing for a gently worn-in look'."""
    # Remove '.txt' suffix if present and map to corresponding URIs
    image_urls = [
        filename_to_uri_map.get(filename.removesuffix(".txt"), None)
        for filename in filenames
    ]

    # Format the inputs for the review agent
    serialized_urls = [
        {
            "type": "input_image",
            "detail": "high",
            "image_url": image_url,
        }
        for image_url in image_urls
        if image_url
    ]

    # Run the agent to get product review results
    return await Runner.run(
        review_agent,
        input=[
            {
                "role": "user",
                "content": serialized_urls + [{"type": "input_text", "text": context}],
            }
        ],
    )