Easy way to log requests and responses to LLM's

[pax95]

Was looking for ways to log request/response from call to LLM's, and found this discussion.
I think it would be great if this would be supported as a configuration property as it is in Langchain4j.

Discussed in #450

^{Originally posted by iAMSagar44 March 15, 2024}
Has anyone managed to find a way to log the requests and responses to/from Open AI using Spring AI.
I have tried various logging settings in the application.properties but no luck.
With Langchain4J, there is a specific property for logging the requests and responses to/from the AI models, but couldn't find anything similar with Spring AI.

[piotrooo]

I handle this using the Logbook logging library.

@Bean
public RestClientCustomizer restClientCustomizer(Logbook logbook) {
    return restClientBuilder -> restClientBuilder.requestInterceptor(new LogbookClientHttpRequestInterceptor(logbook));
}

[pax95]

@piotrooo Thanks for the tip. IMHO it is's maybe a bit to much to introduce such a new dependency just for request/response logging.

[ThomasVitale]

I opened a discussion in #512 about this topic. In the meantime, this is how I solved it. You can see the full examples here: https://github.com/ThomasVitale/concerto-for-java-and-ai/blob/main/mousike/src/main/java/com/thomasvitale/mousike/ai/clients/HttpClientAutoConfiguration.java#L16

    @Bean
    RestClientCustomizer restClientCustomizer(HttpClientProperties httpClientProperties) {
        HttpClientConfig clientConfig = HttpClientConfig.builder()
                .connectTimeout(httpClientProperties.getConnectTimeout())
                .readTimeout(httpClientProperties.getReadTimeout())
                .sslBundle(httpClientProperties.getSslBundle())
                .logRequests(httpClientProperties.isLogRequests())
                .logResponses(httpClientProperties.isLogResponses())
                .build();

        return restClientBuilder -> {
            restClientBuilder
                    .requestFactory(new BufferingClientHttpRequestFactory(
                            ClientHttpRequestFactories.get(ClientHttpRequestFactorySettings.DEFAULTS
                                            .withConnectTimeout(clientConfig.connectTimeout())
                                            .withReadTimeout(clientConfig.readTimeout()))))
                    .requestInterceptors(interceptors -> {
                        if (clientConfig.logRequests() || clientConfig.logResponses()) {
                            interceptors.add(new HttpLoggingInterceptor(clientConfig.logRequests(), clientConfig.logResponses()));
                        }
                    });
        };
    }

[piotrooo]

@piotrooo Thanks for the tip. IMHO it is's maybe a bit to much to introduce such a new dependency just for request/response logging.

As always, it depends. If you want something for development purposes, a @ThomasVitale example is enough. On the other hand, if you want a fully-featured sample with configuration and battle-proven features, use library (e.g. logbook).

[pax95]

Both solutions are viable solutions for something that you would need in the initial development and debug fase.
(I suspect that both solutions would affect other restclients too if you have Functions that uses a custom restClient)
IMHO this should be easier for developers to enable using a configuration property as part of spring-ai.

[markpollack]

I've added some thoughts for discussion here. Certainly improvements in this area need to be made.

[ThomasVitale]

I shared some additional thoughts in #512 (comment)

[rwankar]

I'm using SpringBoot 3.4.3 and Spring AI 1.0.0-SNAPSHOT and the solution @ThomasVitale provided didn't work as the API seems to have changed.

I then tried the solution by @piotrooo and provided my own log implementation. The restClientCustomizer method is called but my logger was not being invoked. I'm not sure why that is.

    @Bean
    public RestClientCustomizer restClientCustomizer(HttpClientProperties httpClientProperties)
    {
        return restClientBuilder -> {
            restClientBuilder.requestInterceptor(new ClientLoggerRequestInterceptor());
        };
    }

After much frustration I landed on this StackOverflow link https://stackoverflow.com/questions/78444230/how-to-change-the-restclient-implementation-for-springai.

The actual logging code is a copy from Dan Vega's article here https://www.danvega.dev/blog/spring-boot-rest-client-logging

Here is my implementation that works. May be it can help someone. If I was making a mistake I'd like to know what it is.

     final RestClient.Builder builder = RestClient.builder()
         .requestInterceptor(new ClientLoggerRequestInterceptor());

     OpenAiApi openAiApi = OpenAiApi.builder()
         .baseUrl("https://api.openai.com")
         .apiKey(ApiKeys.get(ApiKeys.OPENAI_API_KEY))
         .restClientBuilder(builder)
         .build();

    public static class ClientLoggerRequestInterceptor implements ClientHttpRequestInterceptor
    {
        private static final Logger log = LogManager.getLogger(ClientLoggerRequestInterceptor.class);

        @Override
        public ClientHttpResponse intercept(HttpRequest request, byte[] body,
            ClientHttpRequestExecution execution) throws IOException
        {
            logRequest(request, body);
            var response = execution.execute(request, body);
            return logResponse(request, response);
        }

        private void logRequest(HttpRequest request, byte[] body)
        {
            log.info("Request: {} {}", request.getMethod(), request.getURI());
            log.debug(request.getHeaders());
            if (body != null && body.length > 0)
            {
                log.info("Request body: {}", new String(body, StandardCharsets.UTF_8));
            }
        }

        private ClientHttpResponse logResponse(HttpRequest request,
            ClientHttpResponse response) throws IOException
        {
            log.info("Response status: {}", response.getStatusCode());
            log.debug(response.getHeaders());

            byte[] responseBody = response.getBody().readAllBytes();
            if (responseBody.length > 0)
            {
                log.info("Response body: {}",
                    new String(responseBody, StandardCharsets.UTF_8));
            }

            // Return wrapped response to allow reading the body again
            return new BufferingClientHttpResponseWrapper(response, responseBody);
        }
    }

    private static class BufferingClientHttpResponseWrapper implements ClientHttpResponse
    {
        private final ClientHttpResponse response;
        private final byte[] body;

        public BufferingClientHttpResponseWrapper(ClientHttpResponse response,
            byte[] body)
        {
            this.response = response;
            this.body = body;
        }

        @Override
        public InputStream getBody()
        {
            return new ByteArrayInputStream(body);
        }

        // Delegate other methods to wrapped response
        @Override
        public HttpStatusCode getStatusCode() throws IOException
        {
            return response.getStatusCode();
        }

        @Override
        public HttpHeaders getHeaders()
        {
            return response.getHeaders();
        }

        @Override
        public void close()
        {
            response.close();
        }

        @Override
        public String getStatusText() throws IOException
        {
            return response.getStatusText();
        }
    }

[ThomasVitale]

There are currently two ways to get the LLM requests and responses:

• using the SimpleLoggerAdvisor (see https://docs.spring.io/spring-ai/reference/api/chatclient.html#_logging)
• enabling the dedicated instrumentation (see https://docs.spring.io/spring-ai/reference/observability/index.html#_chat_prompt_and_completion_data)