[BUG] The file_search tool is often triggered incorrectly (Assistants API)

[MatthiasKerat]

API Spec link

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/file-search?tabs=python

API Spec version

2024-12-01

Describe the bug

We have enabled the file_search tool for the Assistants API along with other custom tools, such as one for web search for example. However, when using the Azure Assistants API, the file_search tool gets triggered often, even when a web search would be the correct choice for a user query. There are days where everything works well in most of the cases and then there are day where it does almost every time choose the wrong tool.

If we remove the file_search tool, everything works as expected. Similarly, when using the OpenAI Assistants API with file_search enabled, everything also functions correctly (With exactly the same system prompt). But Azure behaves completely differently and unreliably, and it seems that the file_search tool is severely interfering with the correct function selection.

I can't understand why the Azure OpenAI Assistants API performs so much worse than OpenAI's. There are so many other unexplained issues, like the lack of image input support in Azure OpenAI, even though it works perfectly fine in OpenAI's API. Currently, the Azure Assistants API is completely unusable.

Expected behavior

That the function call works just as reliably as in the equivalent OpenAI Assistants API.

Actual behavior

The behavior is described above and should contain enough information. There is no stacktrace or something similar.

Reproduction Steps

Activate file_search and add a custom function for web_search for example, instructing the model to use this tool for current events or questions about people. Then, ask a question about a recent event or a person.

It doesn't happen every time, but in many cases, the file_search tool is triggered instead of the intended web search.

Environment

No response